TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions

In plain words

Mixed traffic is hard to predict. Cars, buses, scooters, rickshaws, bicycles and pedestrians all move differently, and in dense traffic each one reacts to only a few others. Most trajectory models were built for lanes, highways or crowds of walkers, and they treat every neighbour as roughly equally important.

TraPHic takes a different view. It feeds each road user's recent path, speed, local traffic density and size into a small neural network. It pays most attention to the road users in a region in front of the target vehicle, and it learns how much each nearby road user should count, based on what kind of agent it is and how it moves. The output for each target is a short future path, given as a spread of likely positions over time. The authors also release a new dense, heterogeneous traffic dataset recorded in urban Asian cities.

TL;DR

• TraPHic predicts the next few seconds of positions for every road agent in dense, mixed traffic, using a hybrid of LSTMs and convolutional networks §4.
• It learns weights for two interaction sets: road agents in a forward "horizon" region, and nearby neighbours. Both are pooled into grids and encoded with ConvNets §4.1.
• On the authors' TRAF dataset, the full model reports 0.78 m ADE and 2.44 m FDE. The variant without weighted interactions reports 2.73 m and 7.21 m tab. 2.
• On the sparse NGSIM set it is not the best method (S-GAN has lower ADE) tab. 1.
• Checked by reading the public code: its default training entry point does not run as shipped (see Paper vs code).

Problem

• Learned predictors that pool neighbours uniformly (Social-LSTM-style pooling, CS-LSTM) give every interaction the same weight. The authors argue this fails in dense heterogeneous traffic, where the dynamics of, say, a bus and a pedestrian differ strongly §1.
• Many earlier methods were evaluated on sparse or homogeneous data (NGSIM, KITTI), where interactions are few and can be modelled with classical social-force or velocity-obstacle models §1.
• The authors define high-density traffic as more than 100 road agents per km §1.
• The method assumes lane-free driving. CS-LSTM uses lane information, so for the TRAF comparison the authors used a no-lane variant §5.4.
• The model needs agent shapes and sizes. The authors list collecting that information as a limitation §6.
• Positions are converted from image pixels to world coordinates, which needs camera calibration §3.1.

Method

1. Agent state. Each road agent a_i has the state Ω_i (eq. 1) §3.1. It contains t past positions Ψ, their differences ΔΨ (velocity), a traffic concentration value c_i along the track, and the agent's size l_i §3.1. The concentration at a location counts the agents in a box offset by a fixed (δx, δy) from that point §3.1.
2. Neighbourhood and horizon. The neighbourhood N_i is found by nearest-neighbour search inside an elliptical region around the ego agent. The horizon H_i is a smaller semi-elliptical region in front of it, searched the same way §3.1.
3. Embedding. Each agent's state is embedded as e_j = φ(W_l Ω_j + b_l) with an ELU non-linearity (eq. 2) §4.
4. Horizon and neighbour LSTMs. Each embedding goes through a single-layer LSTM, which gives a hidden state h_j(t) (eq. 3) §4. In the horizon layer, the hidden states pass through one more fully connected layer with ELU, giving h_jw §4. The h_jw of all horizon agents are pooled into a horizon map H_i (eq. 5) §4.1. The hidden states of neighbours are pooled directly into a neighbour map N_i (eq. 6) §4.1.
5. Map encoders. Each map goes through its own ConvNet. Each ConvNet has two convolutions, each followed by ELU, and then max-pooling §4§5.2. The outputs are f_hz (horizon) and f_nb (neighbour) §4.
6. Ego branch. The ego agent's embedding passes through a fully connected layer with ELU and a single-layer LSTM, giving h_i §4.
7. Fusion and decoding. The three vectors are concatenated, z = concat(h_i, f_hz, f_nb) (eq. 4) §4. An LSTM decodes z into the prediction for the next τ steps §4.
8. Output and loss. Each future step outputs a bivariate Gaussian with means, standard deviations and a correlation coefficient §4.3. Training minimises the negative log-likelihood summed over future steps (eq. 7) §4.3. The learned weights depend on each agent's shape, dynamic constraints and behaviour §4.1. The learned weights of the horizon layer's linear block are read as the priority given to each horizon interaction §4.3.
9. Implicit constraints. Turning limits are covered implicitly, through the agent size in the state §4.2. Driver behaviour is covered implicitly, through relative and average velocity and acceleration, the concentration difference between consecutive locations, and the distance to neighbours, all placed in the state §4.2. The paper does not label behaviour explicitly.

# TraPHic forward pass for one ego agent  [§4] [eq. 2-6]
# Input:  ego_state (T, d); neighbour_states (N, T, d); horizon_states (H, T, d)
# Output: per future step, a bivariate Gaussian (mu_x, mu_y, sigma_x, sigma_y, rho)

def predict(ego_state, neighbour_states, horizon_states):
    # Embed every agent's state with a linear layer and ELU  [§4] [eq. 2]
    neighbour_emb = elu(linear(neighbour_states))   # (N, T, e)
    horizon_emb = elu(linear(horizon_states))       # (H, T, e)

    # Encode each agent's history with an LSTM  [§4] [eq. 3]
    neighbour_hidden = lstm_neighbour(neighbour_emb)  # (N, hidden)
    horizon_hidden = lstm_horizon(horizon_emb)        # (H, hidden)

    # Horizon agents get one more FC + ELU, then are pooled into a map  [§4.1] [eq. 5]
    horizon_map = pool_to_grid(elu(fc_horizon(horizon_hidden)), horizon_states)

    # Neighbours are pooled directly into a map  [§4.1] [eq. 6]
    neighbour_map = pool_to_grid(neighbour_hidden, neighbour_states)

    # Each map goes through a ConvNet: two convolutions with ELU, then max-pooling  [§4] [§5.2]
    horizon_feature = conv_pool(horizon_map)
    neighbour_feature = conv_pool(neighbour_map)

    # Ego agent: embed, FC with ELU, single-layer LSTM  [§4]
    ego_hidden = lstm_ego(elu(fc_ego(linear(ego_state))))

    # Concatenate the three encodings  [§4] [eq. 4]
    encoding = concat(ego_hidden, horizon_feature, neighbour_feature)

    # Decode a bivariate Gaussian for each future step  [§4.3]
    return decoder(encoding)

Training objective, as written in the paper:

L_i = - \sum_{t+1}^{\tau} \log P\big((x_i^t, y_i^t) \mid \mu_i^t, \sigma_i^t, \rho_i^t\big) eq. 7

Hyperparameters

Results

Author-reported. Metric: ADE and FDE, RMSE in metres after 5 s tab. 1tab. 2. The NGSIM and Beijing values are from tab. 1; the TRAF values are from tab. 2.

Sparse or homogeneous datasets tab. 1

The authors say their method is on par with the state of the art on NGSIM and does not beat it there §5.4.

Dense, heterogeneous TRAF dataset tab. 2

"Original" means the baseline was run with its default settings; "learned" means it was trained on TRAF tab. 2 caption.

Ablation variants §5.3§5.4:

• TraPHic-B: no weighted interactions.
• TraPHic-H_e: no horizon-based weighting; heterogeneous interactions only.
• TraPHic-H_o: no heterogeneous-based weighting; horizon-based only.
• TraPHic: both.

Relative changes the authors state, computed from tab. 2: H_e vs B reduces ADE by 15% and FDE by 20%. H_o vs B reduces ADE by 55% and FDE by 58%. The combined model vs B reduces ADE by 71% and FDE by 66% §5.4. The authors conclude the horizon-based term contributes more than the heterogeneous term §5.4.

The paper's headline claim is a 30% RMSE improvement over prior methods on dense traffic §1fig. 4. The figure caption also gives 30% relative to CS-LSTM fig. 4; the caption does not say which CS-LSTM row this refers to. Against the learned CS-LSTM row, the combined model is 32% lower on ADE (0.78 vs 1.15), a figure derived here from tab. 2.

Dataset. TRAF has 50 videos of dense, heterogeneous traffic, collected in dense urban Asian traffic §1, at 1280 × 720, with front-facing and top-view cameras, labelled with agent positions, IDs and types §5.1tab. 3. Its per-frame agent count is listed as 19.88 in tab. 3. The dataset page is the project page linked from the paper §5.1; the dataset's licence is not stated in the paper, so this card uses only the reported numbers.

Paper vs code

The paper's project page lists a code repository. That link returns 404 at the time of compilation, so the code used here is the first author's rohanchandra30/TrackNPred repository (commit 3cf1dfca), which contains a TraPHic implementation alongside Social-GAN and other code. It has no LICENSE file, so it is cited by path and line only. The search also found a second author-owned repository, rohanchandra30/TraPHic, which is a website stub with no model code.

• Dropout and batch norm. The paper states that the network uses neither §5.2. The default configuration sets dropout 0.5 train_cmd.py. The model has BatchNorm layers traphic.pytraphic.pytraphic.py. Dropout is applied to pooled features traphic.py and to the predicted outputs traphic.py.
• Horizon and heterogeneous inputs are off by default. The default configuration sets ours to False train_cmd.py. In that path the agent input is only a 2-D position traphic.pytraphic.py. The velocity input and the horizon map branch are used only when ours is True traphic.py. The only script found that sets it True is evaluate_old.py evaluate_old.py. So the default run does not use the velocity, concentration or size inputs that Method describes §3.1.
• Epoch count. The paper trains for 16 epochs §5.2 and describes no pretraining. The code's default schedule is 6 pretraining epochs plus 10 training epochs, which sum to 16 train_cmd.pytrajPredEngine.py. The pretraining epochs use a pretrain_loss setting trajPredEngine.py.
• Missing loss keys. train_cmd.py never sets pretrain_loss or train_loss in its parameter dictionary train_cmd.py. The engine reads both keys by direct indexing trajPredEngine.pytrajPredEngine.pytrajPredEngine.py. By reading, the default training run should raise a KeyError on its first batch. This was not run.
• Training data path. train_cmd.py loads model/Prediction/data/TRAF/TRAFTrainSet.npy and TRAFValSet.npy train_cmd.py. The repository contains TrainSet.npy, ValSet.npy and TestSet.npy in that folder, not files with the TRAF prefix.
• Trained weights. Evaluation loads model/Prediction/trained_models/<model>_<name>.tar train_cmd.py. That folder is not in the repository, so no trained weights are available.
• Loss scale. The paper sums the negative log-likelihood over future steps (eq. 7) §4.3. The code's maskedNLL divides the sum by the number of valid steps utils.py. This changes the loss scale, not the minimiser.
• Output parameterisation. The paper says only "bivariate Gaussian" §4.3. The code uses the means as they are, exponentiates the two spreads, and applies tanh to the correlation utils.py.
• Decoder input. The code repeats the encoding for each of the 50 output steps as the decoder LSTM input traphic.py. The paper does not say how the decoder is fed.
• Metric. The paper defines ADE as the RMSE over all predicted positions, and FDE as the RMSE at the final position §5.3. The code takes the square root of a per-step aggregate and then averages over steps trajPredEngine.py. These can differ; the tensor shapes were not traced.
• Units. The code multiplies RMSE by 0.3048 to convert from feet to metres trajPredEngine.py. The paper reports metres and does not say the data is in feet tab. 2.

Pitfalls

• Equation 2 typo. It writes Ω_i for the embedding of agent a_j. The reading that matches the rest of the section is Ω_j §4.
• Map placement is not stated. The paper does not say how agents are placed into the horizon and neighbour grids. The code uses masks built by the data loader traphic.py. Reproducing the paper requires that placement, and this card does not describe the mask construction.
• Neighbour count and geometry. The number of nearest neighbours is not given. The offset (δx, δy) used for concentration is not given. The ellipse and semi-ellipse shapes are only partly specified (radius 2 m, minor axis 1.5 m) §3.1§5.2.
• Camera calibration. Camera parameters and the homography are said to come from standard techniques §3.1. No procedure is given.
• Frame rate and history. The paper gives 3 s of history, but not the frame rate, so frame counts in the code cannot be checked against seconds.
• Real-time claim. The paper calls the method real-time §1, but reports no latency or throughput figures.
• Compute cost. Only the GPU model is given §5.2. Memory and training time are not reported.
• Dataset statistics. The text gives about 13 motorised vehicles, 5 pedestrians and 2 bicycles per frame §5.1. tab. 3 gives a total of 19.88 agents per frame. These are close, but they are not identical.
• Not reported: learning-rate schedule, gradient clipping, weight initialisation, and seeds or variance across runs. Only single results are shown tab. 1tab. 2.
• Sparse data. On NGSIM, S-GAN has the lower ADE. The authors call their method comparable, not best tab. 1§5.4.
• Ablation scope. The ablation is run on TRAF only §5.4.
• Figure and equation numbers. Numbers here are derived from LaTeX source order (fig. 1 to fig. 5, eq. 1 to eq. 7, tab. 1 to tab. 3). They were not checked against a compiled PDF.

Lineage

• Builds on: the social-pooling approach of Social-LSTM, which the paper follows for its prediction scheme §4; and CS-LSTM, which adds convolutional social pooling and which the paper uses as a baseline §5.3. Neither has a card on this wiki.
• Compared against: RNN-ED, Social-LSTM (S-LSTM), Social-GAN (S-GAN) and CS-LSTM, each in an original and a TRAF-trained setting where applicable §5.3tab. 2.
• Superseded by: not known from content on this wiki.