wikillm

Paper · 2025-12-28 · cs.LG, cs.AI, cs.RO

Value-guided action planning with JEPA world models

Trains a JEPA state encoder so that the Euclidean (or quasi-) distance between state embeddings approximates the negative goal-conditioned value function, using an expectile IQL loss. Plans with MPPI-MPC. Author-reported planning accuracy, VF_quasi vs pred_VCReg: wall WS 0.71 vs 0.55, WB 0.96 vs 0.89, maze 0.63 vs 0.54.

World modelGoal-conditioned reinforcement learningModel predictive control (MPC)Self-supervised learningworld-modelsjoint-embedding-predictive-architectureimplicit-q-learningmodel-predictive-controlmppiplanning

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A world model predicts what the world will look like after an action. To pick good actions, a planner imagines several futures and keeps the one that ends closest to the goal. Plain distance between the predicted picture and the goal picture is a poor guide: many paths lead to small local dips, so the planner gets stuck.

This paper changes the space the model thinks in. Each image is mapped to a point, and the straight-line distance between two points is trained to act like a "how many steps to the goal" score, with a sign flip so that closer means better. The training signal comes from offline trajectories, with a reinforcement-learning loss that favours the best paths seen in the data. The planner then uses that score to choose actions. On simple wall and maze tasks, this beats the standard way of training these models, which only predicts the next embedding.

TL;DR

The state encoder of a JEPA world model is trained with an implicit Q-learning (IQL) loss, so that embedding distance approximates the negative goal-reaching value. Planning is MPPI inside model predictive control. On two wall datasets and one maze dataset the authors report higher planning success than JEPA models trained with prediction and regularisation losses. The paper gives no official code, and several details needed to reimplement it are not stated.

Problem

  • JEPA world models are trained to predict future embeddings. The authors aim to improve their planning capability §1.
  • Planning by minimising the distance between a predicted and a goal embedding yields a cost with many local minima, which makes optimisation hard §3.
  • Standard fixes are a VCReg or EMA term against collapse, which do not shape the embedding space for goal-reaching §2.1.
  • Prior work that learns goal-reaching value functions mostly targets policy execution, not planning inside MPC with a JEPA model §3.

Method

The paper learns the state encoder of a JEPA model so that a value function is read from the embedding space. Two parts matter: the value definition and the loss that trains it §3.2.

Value definition §3.2. For states s and goals g, the value is the negative Euclidean distance between their embeddings, V_θ(s, g) = −‖E_θ(s) − E_θ(g)‖₂. The target is the goal-conditioned value V* of the reaching cost C(s, a, g) = 1[s ≠ g], which charges every step not at the goal §3.2.

Loss §3.2 eq. 1. The loss is an expectile-regression TD error on embedding distances. For each trajectory and each goal in the batch, the reward is −1 when the state differs from the goal and the bootstrapped next value is discounted by γ. The next-state term is treated as a constant (stop-gradient). The residual is squared and weighted asymmetrically by τ. The loss is summed over the goals n and the time steps t eq. 1.

Goals §3.2. Two goal types are used: the last state of each training trajectory, and random states drawn from the training batch.

Quasi-distance variant §3.2. The paper also replaces the Euclidean distance with a quasi-distance, following Wang et al. §3.2. The formula of that quasi-distance is not given in this paper. The text only cites the generic form of Wang and Isola (2022). This variant, VF_quasi, is the best result in the paper tab. 2.

Training regimes §3.2.

  • Sep: first train the state encoder alone with L_VF, then train the action encoder and predictor with the prediction loss L_pred.
  • Joint: train all networks together, with the objective L_VF + L_pred.

Baselines §3.1. Two non-value losses for the state encoder are compared: a contrastive loss (successive states as positives, random pairs as negatives) and a regression loss that pushes the distance between successive states towards 1. Table 1 lists the full set of training variants: Contrastive, Regressive, pred_VCReg, pred_EMA, VF, VF_pred, VF_quasi, VF_quasi_pred, VF_VCReg and VF_VCReg_pred tab. 1.

Planning §3 §4.2 §7.2. Planning is model predictive control with an MPPI optimiser. For value-based representations, the planning cost is the learned value. The paper says that minimising it "naturally drives the model toward the goal" §3. The exact per-step cost form is not stated. MPPI settings are in the table under Hyperparameters.

Pseudocode
# Value-function (IQL) loss for one trajectory  [§3.2] [eq. 1]
# Input: states (T+1, obs), goals (N+1, obs)
# Output: scalar loss
def vf_loss(states, goals, gamma, tau):
    # Embed states and goals with the same state encoder  [§3.2]
    z_states = encoder(states)                  # (T+1, d)
    z_goals = encoder(goals)                    # (N+1, d)
    total = 0.0
    for n in range(N + 1):
        # Reward -1 for each step where the state differs from the goal  [eq. 1]
        reward = -(states[:-1] != goals[n])     # (T,)
        # Next-state value, treated as a constant  [§3.2] [eq. 1]
        next_value = stop_gradient(
            value(z_states[1:], z_goals[n]))    # (T,)
        # Current value, V = -distance  [§3.2]
        value_now = value(z_states[:-1], z_goals[n])   # (T,)
        # Expectile loss: |tau - 1[x < 0]| * x^2 on x = TD error  [eq. 1]
        residual = reward + gamma * next_value - value_now
        total += expectile(residual, tau).sum()
    return total


def value(z_a, z_b):
    # Negative Euclidean distance between embeddings  [§3.2]
    return -norm(z_a - z_b, axis=-1)


def expectile(x, tau):
    # Asymmetric squared loss  [eq. 1]
    return abs(tau - (x < 0)) * x ** 2
Pseudocode
# One MPC step with MPPI  [§7.2] [§3]
# Input: current embedding, goal embedding, noise mean and std
# Output: first action of the optimised sequence
def plan_step(z_now, z_goal, mean, sigma, lam, horizon):
    # Sample action perturbations around the current mean  [§7.2]
    noise = normal(0, sigma, (K, horizon, action_dim))   # K samples
    actions = mean + noise                              # (K, H, a)
    # Roll out the learned predictor for every sample  [§7.2]
    z_pred = rollout(predictor, z_now, actions)         # (K, H, d)
    # Score each rollout with the planning cost; exact form not stated  [§3]
    cost = planning_cost(z_pred, z_goal)                # (K,)
    # Exponential weights with temperature lambda  [§7.2]
    weights = softmax(-cost / lam)                      # (K,)
    mean = sum(weights * actions)                       # MPPI update
    return mean[0]                                      # first action only

Key equation: the only numbered equation in the paper is the IQL loss eq. 1. The VCReg, EMA and quasi-distance formulas are not written out in this paper.

Hyperparameters

Values from the appendix §7.2 and the sweep figure fig. 3. "Not stated" means the paper does not give the value.

NameValueWhereNotes
discount γ0.98 (VF); 0.93 (VF_quasi)§7.2 fig. 3Tuned on a separate WS dataset §7.3
expectile τ0.80 (VF); 0.60 (VF_quasi)§7.2 fig. 3Tuned as above; the paper says τ near 1 is theoretically preferred but unstable §7.3
representation size512, flat vector§7.2
state encoder2.2M parameters; convolutions with residual connections§7.2Layer counts and widths not stated
predictorMLP, 1.3M parameters§7.2Layer count and width not stated
action encoderidentity§7.2Actions and state representations concatenated before the predictor §7.2
training segment length16§7.2Trajectories are cut into segments of length 16
optimiserAdam, base learning rate 0.0028, cosine schedule§7.2Batch size, number of steps, warm-up, weight decay: not stated
VCReg termalong batch (wall); along batch and time (maze)§7.2Loss weight: not stated
MPPI samples K2000 (wall); 500 (maze)§7.2
MPPI noisemean 0, std 12 (wall); mean 0, std 5 (maze)§7.2
MPPI temperature λ0.005 (wall); 0.0025 (maze)§7.2
planning horizon96 for 200 steps (WS); 64 for 64 steps (WB); 100 for 200 steps (maze)§7.2Re-planning per step is implied by "MPC" §7.2
joint objective weightL_VF + L_pred (weight 1 each implied)§3.2Weighting not stated explicitly
evaluation instances200 (wall); 80 (maze)§4.2Success threshold not stated
number of seedsnot stated

Datasets

  • Wall, observations 64×64 with 2 channels (agent and walls) §7.1.1. The wall and door positions are random per instance §4.1. Training data are 1000 trajectories of length 64, half of which pass through the door. Actions are displacement vectors: the direction is drawn with von Mises noise of concentration 5 §7.1.1.
    • WS: action norm Gaussian with mean 1 pixel, std 0.4, clipped to [0.2, 1.8].
    • WB: action norm Gaussian with mean 2 pixels, std 0.8, clipped to [0.4, 3.6].
    • Example trajectories are in fig. 1.
  • Maze, from the PointMaze setup of Sobal et al. §7.1.2. A 4×4 grid with 50 to 60% of the squares selected as walls, observations 64×64 with 3 colour channels §7.1.2. Actions are target speeds, and the environment computes the force needed to reach them §7.1.2. Training uses 1000 trajectories of length 101 §7.1.2, drawn from five layouts that differ from the evaluation layouts §7.1.2. Evaluation start and goal are at least 3 cells apart §7.1.2. The velocity is an input to the encoders §4.1. Example states are in fig. 2.

Results

Author-reported. Setup: planning success rate, the proportion of successful plans for random start and goal pairs, using MPC with MPPI §4.2. Evaluation uses 200 wall instances and 80 maze instances §4.2. Table 2 gives the full grid. One wall instance changes the score by 0.005.

MethodSepWSWBMazeSource
Contrastiveyes0.490.590.50tab. 1 tab. 2
Regressiveyes0.540.570.46tab. 1 tab. 2
pred_VCRegno0.550.890.54tab. 1 tab. 2
pred_EMAno0.460.430.04tab. 1 tab. 2
VFyes0.630.940.49tab. 1 tab. 2
VF_predno0.550.750.49tab. 1 tab. 2
VF_quasiyes0.710.960.63tab. 1 tab. 2
VF_quasi_predno0.610.850.43tab. 1 tab. 2
VF_VCRegyes0.490.750.39tab. 1 tab. 2
VF_VCReg_predno0.470.670.39tab. 1 tab. 2

Headline claims in the text §4.2:

  • The IQL-based methods give better guidance and better results than the intuitive or prediction-based baselines §4.2.
  • VF_quasi beats VF on every column, "even when the theoretical value function is symmetric" §4.2.
  • Adding a prediction loss to the IQL loss (Joint or VF_pred) does worse than IQL alone §4.2.
  • Adding a VCReg term to the IQL loss performs poorly §4.2.
  • Hyperparameter sweeps are in fig. 3. The text gives no numbers for them.
  • A two-encoder variant (pred_VCReg for prediction, VF for cost) scores 0.60 on WS and "did not improve" planning §7.3. No table is given for it.

Paper vs code

No official code. The arXiv page lists no code link. GitHub searches for "value-guided JEPA" and "JEPA IQL planning" returned nothing. The search "JEPA world model planning" returned repositories by other groups only. The paper says its experiments build on an implementation from Sobal et al. §7.2. That is a different paper's code and is not used here.

Pitfalls

  • Loss normalisation. Eq. 1 sums over goals and time steps, but the text calls it the "mean IQL loss" §3.2. The normalisation changes the effective learning rate. It is not stated which the authors used.
  • Equality test. The reward uses 1[s_t ≠ g_n] eq. 1. For image states the paper does not say how equality is decided. Goals are either trajectory end states or batch samples §3.2.
  • Stop-gradient, not a target network. The bar on the next-state value means stop-gradient §3.2. No separate target network or EMA is described for this term. Take the target from the same encoder, detached.
  • Quasi-distance formula missing. VF_quasi needs the quasi-distance of Wang and Isola (2022), whose formula is not in this paper §3.2. Get it from that source. Results for VF_quasi cannot be reproduced without it.
  • Table 1 vs text for the regressive baseline. The text describes the regression loss alone §3.1. Table 1 lists L_regressive together with L_VCReg tab. 1. The VCReg weight is not stated.
  • EMA baseline. The EMA procedure has no decay rate or update schedule in this paper tab. 1 §2.1.
  • Planning cost unstated. The paper says the planning cost is the learned value §3. It does not give the per-step form, whether costs are summed over the horizon or taken at the end, or the MPC re-planning details. The "planning success" threshold is also not stated §4.2.
  • Sample size and seeds. 200 and 80 instances §4.2. Each instance is worth 0.005 (wall) or 0.0125 (maze). Differences of 0.02 to 0.05 on the wall are 4 to 10 instances. No confidence intervals or seed counts are reported. Results look like single runs.
  • Tuning data. γ and τ were tuned on a separate WS dataset §7.3. The sweep figure is the only evidence fig. 3. The τ sweep is shown at fixed γ, the γ sweep at τ = 0.7 fig. 3.
  • Segment length vs trajectory length. Training segments have length 16 §7.2. Wall trajectories have length 64 and maze trajectories 101 §7.1.1 §7.1.2. Eq. 1 uses T for the trajectory length, and the paper never says whether T is 16 in training. Do not assume.
  • Distant states. The authors report that value estimates for distant state pairs are inaccurate. Sparse sampling of distant triplets and small gradients far from the goal are the given reasons §5. They suggest a hierarchy of representations, but do not test one §5.
  • Dataset effect. WB scores exceed WS for 9 of the 10 methods in Table 2. The exception is pred_EMA (0.43 vs 0.46) tab. 2. The paper's explanation is that single WB trajectories cover more of the environment §4.2.
  • Stochastic environments. The authors expect IQL to be biased in random environments. They did not test this §6.
  • Data generation. Wall trajectories use random directions with von Mises noise, not Gaussian action noise. The authors say Gaussian noise would concentrate trajectories in a small region of the environment §7.1.1. Copy this generator to match the data.

Lineage

  • Builds on: the JEPA framework of LeCun (2022), with planning setup from Sobal et al. (2025, arXiv:2502.14819). The paper's experiments build on Sobal et al.'s implementation §7.2. The value-based representation idea follows HILP (Park et al. 2024, arXiv:2402.15567). IQL is from Kostrikov et al. (2021, arXiv:2110.06169). The quasi-distance is from Wang et al. (2023, arXiv:2304.01203) and Wang and Isola (2022) §3 §3.2.
  • Compared against: Contrastive and Regressive losses §3.1; prediction losses with VCReg or EMA collapse prevention, as in Sobal et al. (2025), Assran et al. (2023, arXiv:2301.08243) and Bardes et al. (2024, arXiv:2404.08471) §2.1 tab. 2.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “Value-guided action planning with JEPA world models” by Matthieu Destrade, Oumayma Bounou, Quentin Le Lidec, Jean Ponce, Yann LeCun (arXiv:2601.00844), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.