Remember Intentions: Retrospective-Memory-based Trajectory Prediction

In plain words

Predicting where a pedestrian or a basketball player will go next is hard to do from a single fixed model, because a model that learns from every past example blends all of them together. MemoNet takes a different route: it keeps a library of past situations, each paired with where that agent ended up, and looks up the entries that most resemble the current one. The prediction is built from those specific matches rather than from a blended set of weights.

Because the lookup is explicit, the system can point back to the stored cases it used. That makes its predictions easier to inspect, which the authors see as important where wrong guesses are costly, such as driving. The library is pruned so that near-duplicate cases are dropped, and the lookup itself is learned rather than a fixed similarity measure.

The prediction runs in two steps. First the model guesses the agent's destination, and then a second model fills in the whole path that leads there. Splitting the task into two simpler problems is the authors' argument for more accurate predictions.

TL;DR

MemoNet is an instance-based trajectory predictor: it retrieves similar training cases from a filtered memory bank, decodes destinations from them, and clusters the results into K candidate destinations. A trajectory fulfilment model then produces full paths. The authors report lower final-displacement error than the previous best on SDD, ETH-UCY and NBA. The official code covers SDD and ETH-UCY; it has no NBA code, and several settings differ from the paper.

Problem

Parameter-based trajectory predictors fit every training pair into one set of weights. The authors argue that this gives no clear link between a prediction and the examples behind it, and that many unrelated training pairs can disturb the prediction for the current scene §1. They add that this lack of interpretability matters in safety-sensitive uses such as autonomous driving §1. The task is to predict an agent's future path from its own past and its neighbours' pasts, and the authors frame the intention as the destination, which carries most of the movement pattern §3. The evaluated datasets cover pedestrians (SDD, ETH-UCY) and basketball players (NBA) §6.1. The authors also show that a prediction can be traced back to the stored instances it used §6.4.

Method

The method has four designs §1: (i) a joint reconstruction model that creates matching past and intention features; (ii) a training-free memory filter; (iii) a learned addresser that scores memory addresses; (iv) an intention clustering step for diverse outputs. Trajectory fulfilment is a separate model that is trained last §5.2.

1. Frame the task. Predict the target agent's future positions from its past and its neighbours' pasts. The destination is the intention §3.
2. Feature learning (stage 1). Train a social encoder for the past, an intention encoder for the destination, and a shared decoder that reconstructs both the past and the destination from the concatenated features §4.1. The joint loss is an L2 term on the past plus alpha times an L2 term on the destination §4.1. The past feature is 128-dimensional and the intention feature is 64-dimensional §6.2.
3. Memory bank. Freeze the encoders, run every training sample through them, and store the past features in M_past and the intention features in M_int at the same address §4.1. The memory is then filtered: an entry is redundant if an entry already kept is within theta_past in start position and within theta_int in destination eq. 1. Algorithm 1 applies that rule sample by sample alg. 1.
4. Addresser. Two MLPs project the query past feature and each memory key §4.2. The score for address i is the cosine similarity of the projected features. Scores are trained against a soft pseudo label derived from how far each address's decoded destination lies from the true destination §4.2.
5. Intention clustering. Retrieve the L highest-scoring addresses, decode each into an anchor destination, and run K-means to reduce the L anchors to K final destinations §4.3 §6.4.
6. Inference. Encode the query past, score the memory, keep the top L, and decode and cluster as above §4.4.
7. Trajectory fulfilment. Encode the past with social influence, embed the predicted destination with an MLP, concatenate the two, and decode the full future and the reconstructed past §5.1. A fixed 4-stage schedule trains the modules in turn §5.2.

# Filter the memory bank  [§4.1] [alg. 1]
# starts, dests: (N, 2); past feats (N, 128); dest feats (N, 64)
def filter_memory(starts, dests, theta_past, theta_int):
    # No kept entries at the start
    kept = []
    # The paper picks samples at random; the code scans in data order
    for i in samples:
        # Redundant if a kept entry is close in start and in dest  [eq. 1]
        redundant = any(
            norm(starts[i] - starts[j]) <= theta_past
            and norm(dests[i] - dests[j]) <= theta_int
            for j in kept
        )
        # Keep only the entries that are not redundant  [alg. 1]
        if not redundant:
            kept.append(i)
    # Addresses of the filtered memory  [§4.1]
    return kept

# Stage 1: joint reconstruction  [§4.1]
# past: (B, 8, 2) rel. to last frame [code: trainer/trainer_AIO.py#L118]
# dest: (B, 2)
def joint_reconstruction_loss(past, neighbours, dest, alpha):
    # Past feature with social influence  [§4.1]
    past_feat = social_encoder(past, neighbours)     # (B, 64) + (B, 64)
    # Destination feature  [§4.1]
    dest_feat = intention_encoder(dest)              # (B, 64)
    # One decoder rebuilds the past and the destination  [§4.1]
    past_hat, dest_hat = decoder(concat(past_feat, dest_feat))
    # L_rec: squared L2 on past + alpha * squared L2 on dest  [§4.1]
    loss = sum_sq(past_hat - past) + alpha * sum_sq(dest_hat - dest)
    return loss

# Addresser scores for one query  [§4.2]
def addressing_scores(query, memory_keys):
    # Two MLPs map the query and each key into a matching space  [§4.2]
    query_proj = mlp_query(query)                    # (128,) -> (128,)
    keys_proj = mlp_key(memory_keys)                 # (M, 128) -> (M, 128)
    # Cosine similarity of the projected query with every key  [§4.2]
    scores = cosine(query_proj, keys_proj)           # (M,)
    return scores

# Addresser training with a pseudo label  [§4.2]
def addresser_loss(scores, dest_true, dests_decoded, d_t):
    # Distance from each decoded memory destination to the true one  [§4.2]
    dist = norm(dests_decoded - dest_true)           # (M,)
    # Soft target: 1 at zero error, 0 at or beyond d_t  [§4.2]
    target = max(0, (d_t - dist) / d_t)              # (M,)
    # Squared error summed over all addresses  [§4.2]
    return sum((scores - target) ** 2)

# Inference: retrieve, decode, cluster  [§4.4] [§4.3]
def predict_destinations(past, neighbours, memory, L, K):
    # Encode the query past with social influence  [§4.4]
    query = social_encoder(past, neighbours)
    # Score every memory address with the addresser  [§4.2]
    scores = addressing_scores(query, memory.keys)
    # Keep the L best addresses: L = 120 (SDD), 320 (ETH-UCY)  [§6.2]
    top = argsort(scores, descending=True)[:L]
    # One anchor per retrieved address; decoder input see Pitfalls  [§4.4]
    anchors = [decode(query, memory.intention[i]) for i in top]
    # Cluster the L anchors into K final destinations  [§4.3]
    return kmeans(anchors, K)

# Trajectory fulfilment for one destination  [§5.1]
def fulfil_trajectory(past, neighbours, dest_hat):
    # Fulfilment encoder over the past with social influence  [§5.1]
    past_feat = fulfil_encoder(past, neighbours)
    # MLP embedding of the predicted destination, concatenated  [§5.1]
    feat = concat(past_feat, mlp_dest(dest_hat))
    # Decode the full future and reconstruct the past  [§5.1]
    future_hat, past_hat = fulfil_decoder(feat)
    # L_traj: squared L2 on past + beta * squared L2 on future  [§5.1]
    loss = sum_sq(past_hat - past) + beta * sum_sq(future_hat - future)
    return loss

The key equations, unnumbered in the paper except eq. 1:

\mathcal{L}_{\rm rec} = \|\widehat{\mathbf{X}}-\mathbf{X}\|_2^2 + \alpha\,\|\widehat{\mathbf{y}}^{T_f}-\mathbf{y}^{T_f}\|_2^2 \quad [\S 4.1]

\|\mathbf{x}_i-\mathbf{x}_j\|_2 \le \theta_{\rm past}, \quad \|\mathbf{y}_i-\mathbf{y}_j\|_2 \le \theta_{\rm int} \quad [\text{eq. 1}]

s_i = \frac{\mathcal{F}_q(\mathbf{q})\,\mathcal{F}_k(\mathbf{k}_i)^{\top}}{\|\mathcal{F}_q(\mathbf{q})\|_2\,\|\mathcal{F}_k(\mathbf{k}_i)\|_2} \quad [\S 4.2]

\mathcal{L}_{\rm Addr} = \sum_{i=1}^{M}\Big(s_i-\max\big(0,\tfrac{d_T-d_i}{d_T}\big)\Big)^2 \quad [\S 4.2]

\mathcal{L}_{\rm traj} = \|\widehat{\mathbf{X}}_{\rm full}-\mathbf{X}\|_2^2+\beta\,\|\widehat{\mathbf{Y}}-\mathbf{Y}\|_2^2 \quad [\S 5.1]

Training order §5.2: (1) feature learning with the joint loss; (2) freeze the encoders and build and filter the memory banks; (3) freeze the memory and train the addresser; (4) freeze MemoNet and train the fulfilment model.

Hyperparameters

Results

Author-reported. SDD: 4.8 s horizon, minADE_20 / minFDE_20 in pixels, lower is better tab. 1. ETH-UCY: minADE_20 / minFDE_20 in meters, average row used for headline tab. 2. NBA: 4.0 s horizon, meters tab. 3. Table numbers follow the source order of the float environments and were not checked against the compiled PDF. Bold and underline marks from the paper are omitted.

SDD, tab. 1 (4.8 s):

ETH-UCY, tab. 2 (meters):

NBA, tab. 3 (4.0 s, meters):

Headline relative FDE gains, as the paper states them: 20.3% on SDD (15.88 to 12.66), 10.2% on ETH-UCY average (0.39 to 0.35, from Agentformer), 28.3% on NBA (2.05 to 1.47, from NMMP). The paper also reports 28.7% (SDD) and 46.2% (ETH-UCY) over MANTRA §2; these match the table values.

Ablation of components, SDD and ETH-UCY, tab. 4 (cosine = non-learned addresser; ✓ = component on):

Filter threshold on SDD, tab. 5 (theta_past = theta_int):

The paper also varies L on SDD: performance is best at L = 120 and degrades at both smaller and larger values §6.5.

Paper vs code

Code paths are relative to the repository at code_commit. The repository has no LICENSE file; it is cited, never quoted. I searched the paper's footnote link (the authors' GitHub organisation), which is the official repository. The paper's supplementary material is referenced in §6.2 but is not in the source archive, so details it may hold were not checked.

• Optimiser. Paper: SGD §6.2. Code: Adam trainer_AIO.py.
• Addresser learning rate. Paper: 1e-4 §6.2. Code: warm-up uses the 1e-3 default train_MemoNet.py; fine-tuning hard-codes 1e-6 trainer_AIO.py, which matches the paper's fine-tune rate.
• Candidate pre-filter. Paper: the addresser searches the whole memory §4.2 §4.4. Code: first keeps the top 200 addresses by raw cosine similarity, then the learned addresser ranks only those model_AIO.py model_AIO.py.
• Decoder input at inference. Paper: each retrieved intention feature is decoded, without saying which past feature goes with it §4.4. Code: the query's own past feature is concatenated with each retrieved intention feature model_test_trajectory_res.py model_AIO.py.
• Decoder structure. Paper: one decoder reconstructs the past and the destination §4.1. Code: a second stage decodes the residual between the past and its reconstruction and adds it to the first destination model_AIO.py.
• Filter threshold. Paper: theta = 1 on SDD §6.2. Code's training call uses 0.5 trainer_AIO.py. Files are saved with the thresholds and size in the name model_AIO.py.
• Filter order and rule. Paper: random order, invariant to permutation §4.1 alg. 1. Code: sequential data order, greedy model_AIO.py. Code also tests with a strict threshold and an integer count model_AIO.py.
• Test file names. The reproduction model loads sdd_social_filter_past.pt model_test_trajectory_res.py, which the training code never writes (it writes threshold-and-size names) model_AIO.py. Rename or regenerate before running the reproduction.
• Addresser pseudo label. Paper: uses the decoded destination of each memory instance §4.2. Code: uses the stored true destination of each memory instance model_AIO.py loaded from the saved trajectories model_AIO.py. Those saved trajectories are the raw batch coordinates model_AIO.py, while the query destination is relative to the last observed point trainer_AIO.py. The two frames may not match; I did not run this to confirm.
• Addresser loss. Paper: sums over all M addresses, no mask §4.2. Code: masks pairs whose distance exceeds 80, uses 80 as d_T, and divides by the number of unmasked pairs trainer_AIO.py.
• Fulfilment training conditioning. Paper: fulfilment conditioned on the predicted intention §5.1. Code: trains on the destination from the parametric intention head plus Gaussian noise of std 5 model_AIO.py. Inference uses the memory-retrieved clusters model_test_trajectory_res.py model_test_trajectory_res.py.
• Data augmentation. Code: SDD training adds time-reversed tracklets, which doubles the set sddloader.py. Not in the paper.
• Gradient clipping. Code: global norm 1.0 trainer_AIO.py. Not in the paper.
• Model selection. Code: every 5 epochs the trainer evaluates on the test set and keeps the best checkpoint trainer_AIO.py evaluations.py. The paper is silent on selection.
• Oracle best-of-K. Code: the evaluation picks, per sample, the candidate closest to the ground truth at the last frame test_final_trajectory.py. This is the paper's minADE_K / minFDE_K definition §6.3, so it is not deployable as a single prediction.
• Embedding default. test_MemoNet.py defaults the embedding to 24, but the reproduction model hard-codes 64 test_MemoNet.py model_test_trajectory_res.py. The argument has no effect.
• Batch size. --batch_size 32 is unused; the loader uses 512-sample groups train_MemoNet.py train_MemoNet.py.
• NBA. The paper reports NBA results tab. 3. The repository has no NBA training or evaluation script; sddloader.py has only an unused helper sddloader.py. The NBA numbers cannot be checked here.
• ETH-UCY is a separate pipeline. Per-scene configs set L differently from the paper's single L = 320: eth 245 eth.yml, hotel 270 hotel.yml, zara1 240 zara1.yml, zara2 280 zara2.yml, univ 370 univ.yml. The memory threshold is 0.05, not 0.02 model_train_trajectory.py. The eth config uses LR 5e-4 eth.yml, and the config turns on rotation and scale augmentation eth.yml, neither in the paper. The ETH filter subtracts the threshold from the distance before comparing model_train_trajectory.py, and reads self.t_p and self.t_f, which no ETH file assigns that I could find.
• ETH README. It links pretrained files but gives no result numbers. I did not run the ETH pipeline, so the ETH-UCY numbers are unverified.
• Authors' reproduction. The README reports Test FDE_48s: 12.66 and Test ADE: 8.56 for SDD README.md. That output is the authors' own; I did not rerun it.

Pitfalls

• Pixel units. The SDD thresholds of 1 and the addresser's d_T of 80 are in dataset pixels; the paper's tables say pixels for SDD tab. 1. Keep the frame and unit when porting to another dataset.
• Filter is greedy and order-dependent. Kept size depends on the scan order, so the permutation-invariance claim §4.1 does not hold for the code. Expect the memory size to change with data order.
• Filter thresholds are fragile. Storage and accuracy both move with theta tab. 5. Theta 0 keeps about 18k entries; theta 50 keeps 604 and loses about 13 points of FDE.
• Decoder input at inference is unspecified. Pick the query's past feature as the code does model_AIO.py, or the paper's intended design if it can be confirmed. The choice changes every prediction.
• Two-stage training mismatch. Fulfilment is trained on a destination from the intention head with noise, and tested on memory-retrieved clusters. Expect a distribution gap model_AIO.py.
• Test-set checkpointing makes the reported numbers optimistic trainer_AIO.py. Re-split before comparing.
• Oracle metrics. minADE_20 and minFDE_20 choose the best of 20 with ground truth. Do not compare them with single-trajectory metrics.
• Cost per query. The paper reports 18.03 ms per sample on an RTX 3090 §6.5. The inference path scores the whole memory, keeps 200 candidates, runs 120 decoder passes, then runs K-means with 10 iterations per sample model_test_trajectory_res.py. Plan for that cost.
• K-means is randomly seeded and re-seeded on empty clusters model_AIO.py. Fix the seed for repeatable outputs.
• Memory files are external. Pretrained memory and weights come from a Google Drive link with no checksum README.md. Regenerating them with generate_memory needs the PECNet-format pickles, which are not in the repo sddloader.py.
• Lint and reuse. No LICENSE in the code repo: cite it, and do not copy code.

Lineage

• Builds on: retrospective memory and memory networks. The paper names Neural Turing Machines, end-to-end memory networks and differentiable neural computers as the memory-network background §2. None has a card here.
• Compared against: MANTRA, which also uses a memory for single-agent prediction; MemoNet changes the stored item to intentions, makes the addresser learned and adds clustering §2. Also Social-GAN, Social-STGCNN, Trajectron++, SOPHIE, NMMP, EvolveGraph, CF-VAE, PECNet, STAR, STGAT, Agentformer, Transformer-TF, NRI and Social-LSTM tab. 1 tab. 2 tab. 3. None has a card here.
• Superseded by: none known in this wiki.