Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning

In plain words

Reaching a far-away goal in a maze is hard for a learning agent. Earlier hierarchical methods make it plan a path of exact waypoints, and those waypoints are noisy when the goal is far.

This method instead cuts the space into a fixed set of learned zones, called landmarks. A zone is a region of the map that the agent should treat as one piece. A small network learns these zones from recorded trajectories, so that places visited close together in time land in the same zone. A transformer then writes a route as a short list of zones, the way a road trip is described by cities rather than by coordinates. Two learned controllers do the walking: one moves the agent from zone to zone along the route, and the other handles the last stretch to the exact goal.

The payoff is that planning becomes choosing from a small set of discrete labels, and the agent can join pieces of different recorded trips at shared zones. The authors report this helps most on very large mazes.

TL;DR

• QPHIL is a hierarchical offline goal-reaching method for navigation: discrete landmark planning plus two low-level IQL policies §4.
• A VQ-VAE with reconstruction, commitment and temporal triplet losses maps states to landmark tokens §4.1.
• A transformer predicts the landmark sequence, trained with teacher forcing and optional trajectory stitching §4.2.
• Author-reported: competitive on AntMaze medium, best on AntMaze-Ultra (70.3% diverse) and up to 50% success on the new AntMaze-Extreme tab. 1 fig. 5.
• Implementation details are fairly complete, but several ambiguities and table/text conflicts are listed under Pitfalls.

Problem

Offline goal-conditioned RL learns from fixed demonstration trajectories, without interacting with the environment during training §3. With sparse goal rewards, value estimates become noisy as the goal gets farther away, which degrades the advantage weights that IQL-style policies use §3 app. A. HIQL mitigates this with two levels of policies, but its high-level policy must output continuous subgoals whose noise grows with the subgoal horizon k; a larger k hurts the low-level policy, a smaller k hurts the high-level one §3 app. A.

The method targets long-distance navigation where a trajectory can only be formed by stitching recorded pieces together §5.3. It assumes states can be grouped into zones such that the low-level policy can reach any state of a zone from any other state of that zone; the paper states this as an assumption §4.2.

Method

Overview. QPHIL has four learned parts §4:

1. A state quantizer φ that maps each state to one of k landmark tokens §4.1.
2. A plan generator π_plan, a transformer that produces a sequence of landmark tokens for a given goal, auto-regressively until an end token §4.2.
3. A landmark policy π_landmark(a | s, ω) that drives the agent toward the area of a landmark token ω §4.3.
4. A goal policy π_goal(a | s, g) that drives the agent to the exact goal, used after the plan is exhausted §4.3.

Step 1: Tokenizer. φ(s) = argmin over ω of Δ(g(s), z(ω)), where g is an encoder into a latent space, z is a learned codebook of k embeddings, and Δ is the Euclidean distance §4.1. Training minimises a weighted sum of three losses: a reconstruction loss (decode the code back to the input), a commitment loss (stop-gradient terms that pull encoder outputs to their codes), and a triplet-margin contrastive loss over temporal neighbours §4.1. The contrastive term pushes states within a temporal window δ of each other to receive close latents, and pushes states outside that window apart §4.1. The contrastive loss is motivated by thin walls, where spatially close states belong to different situations §4.1 fig. 3.

Step 2: Sequence compression. Once trained, every dataset state becomes a token, and trajectories contain runs of repeated tokens. The post-processing η removes consecutive repetitions, so "1 1 1 2 2 3 3 3 4 4" becomes "1 2 3 4" §4.2.

Step 3: Planner. π_plan(ω_{s+t} | φ(g), history) is a softmax over the tokens, produced by a transformer h_θ from the goal token and the full compressed history; the authors state that conditioning on full history, not only φ(s_t), is useful §4.2. Training is teacher forcing: for each token change in a trajectory, a goal is sampled from the future compressed tokens of that trajectory and the next token is predicted §4.2.

Step 4: Stitching augmentation (optional). If two trajectories visit the same landmark at times t and t′, and the same second landmark at times l and l′, the two spliced trajectories (head of one, tail of the other, at each shared pair) are added to the dataset §4.2. The authors state that the low-level policy is assumed able to move anywhere inside one landmark, which is what makes the splice valid §4.2. The paper describes this as a cheap alternative to offline RL fine-tuning of the planner §4.2. The stitching figure is in the appendix app. B.6.

Step 5: Landmark policy. The landmark policy is trained with IQL-style value learning on relabelled transitions, where the goal is the next different landmark along the trajectory §4.3. Its value function V_landmark is trained on the relabelled set, then the policy is extracted with advantage-weighted regression (AWR) §4.3 eq. 2.

Step 6: Goal policy. The goal policy uses a standard goal-conditioned value network V_goal on raw states and goals, trained with the HIQL-style loss eq. 1, and the policy is extracted with AWR eq. 2 §4.3. It acts on raw, non-quantized goals, for precise final navigation §4.3.

Step 7: Inference (open-loop). At episode start, sample one plan. Follow landmark policies, dropping each landmark when the agent's token equals it. When the plan is empty, switch to the goal policy alg. 1 §4. Re-planning from the current token is optional; the authors' main results use the open-loop version app. D §5.

# Navigation with QPHIL, open-loop  [alg. 1]
# Input: start state, goal state, landmark plan from the planner  [alg. 1]

def navigate(start, goal, max_steps):
    # Plan once at the episode start: tokens, then an end token  [alg. 1]
    plan = sample_plan(phi(goal), [phi(start)])
    state = start
    # Maximum number of steps is not stated in the paper  [alg. 1]
    for step_index in range(max_steps):
        # Follow the next landmark while the plan still has one  [alg. 1]
        if plan[0] != END_TOKEN:
            action = landmark_policy(state, plan[0])
        # Otherwise navigate to the exact goal  [alg. 1]
        else:
            action = goal_policy(state, goal)
        state = environment_step(action)
        # Drop a landmark once its token is reached  [alg. 1]
        if plan[0] != END_TOKEN and phi(state) == plan[0]:
            plan.pop(0)
        # Stop once the goal is reached  [alg. 1]
        elif plan[0] == END_TOKEN and in_goal(state, goal):
            return state
    return state

Tokenizer training objective, as written in §4.1 (the paper writes the three losses as expectations over the dataset):

# Tokenizer loss  [§4.1]
# Input: batch of states, time window delta, loss weights  [§4.1] [app. B.2]

def tokenizer_loss(states, anchors, positives, negatives):
    # Encode, then quantize to the nearest codebook entry  [§4.1]
    latents = encoder(states)
    codes = codebook[nearest_token(latents)]
    # Reconstruction: decode the code and compare with the input  [§4.1]
    reconstruction = squared_error(decoder(codes), states)
    # Commitment: move encoder outputs toward their codes  [§4.1]
    commitment = squared_error(latents, stop_gradient(codes))
    commitment += beta * squared_error(stop_gradient(latents), codes)
    # Triplet margin: temporal neighbours closer than distant states  [§4.1]
    contrastive = max(
        distance(anchors, positives) - distance(anchors, negatives),
        0,
    )
    return reconstruction + alpha_1 * commitment + alpha_2 * contrastive

Compression and planner training:

# Compression eta: drop consecutive repeated tokens  [§4.2]
# Example: 1 1 1 2 2 3 3 3 4 4 becomes 1 2 3 4  [§4.2]

def compress(tokens):
    # Keep a token only if it differs from the token before it  [§4.2]
    kept = []
    for token in tokens:
        if not kept or token != kept[-1]:
            kept.append(token)
    return kept

# Planner loss, teacher forcing  [§4.2]
# Trains on every token change along one trajectory tau

def planner_loss(tau, goals):
    total = 0
    for t in range(len(tau) - 1):
        # Only at token changes, phi(tau_t) != phi(tau_t+1)  [§4.2]
        if phi(tau[t]) == phi(tau[t + 1]):
            continue
        # Goal: one whose token is in the future compressed sequence [§4.2]
        goal = sample_goal(goals, future_tokens(tau, t))
        # Predict the next token from the goal and compressed history  [§4.2]
        history = compress(tokens_up_to(tau, t))
        log_probs = log_softmax(planner(phi(goal), history))
        total += log_probs[phi(tau[t + 1])]
    return -total

Trajectory stitching, as stated in the text:

# Trajectory stitching  [§4.2] [app. B.6]
# Requires phi(tau[t]) == phi(other[t2]) and phi(tau[l]) == phi(other[l2])

def stitch(tau, other, t, t2, l, l2):
    # Head of tau, then tail of other, up to index l2  [§4.2]
    first = tau[0 : t + 1] + other[t2 + 1 : l2 + 1]
    # Head of other, then tail of tau, up to index l  [§4.2]
    second = other[0 : t2 + 1] + tau[t + 1 : l + 1]
    return first, second

Sequence cleanup after tokenization and stitching (the compiled paper's Algorithm 2, restated):

# Sequence cleanup  [alg. 2] [app. B.3]

def clean(sequences):
    kept = []
    seen = set()
    for seq in sequences:
        # Drop tokens already seen in the last 4 steps  [alg. 2] [app. B.3]
        filtered = drop_recent_repeats(seq, window=4)
        # Cut each cycle at its first repeated token  [alg. 2] [app. B.3]
        # Exact truncation point is ambiguous, see Pitfalls
        filtered = cut_cycles(filtered)
        # Keep each distinct sequence once  [alg. 2] [app. B.3]
        key = tuple(filtered)
        if key not in seen:
            seen.add(key)
            kept.append(filtered)
    return kept

Landmark relabelling and advantage weights:

# Next-landmark goal for pi_landmark  [§4.3]

def next_landmark_goal(tau, t):
    # The next landmark is the token of the first later state  [§4.3]
    # whose token differs from the current one
    for l in range(t + 1, len(tau)):
        if phi(tau[l]) != phi(tau[t]):
            return phi(tau[l])
    # Case with no later landmark is not stated in the paper  [§4.3]

# AWR weight for policy extraction  [§4.3] [eq. 2] [app. B.2]

def awr_weight(value_next, value_now, inverse_temperature):
    # Exponential advantage weight, clipped at 100  [app. B.2]
    advantage = value_next - value_now
    return min(exp(inverse_temperature * advantage), 100.0)

Hyperparameters

Values are the ones the paper states. Anything not listed is "not stated". The paper gives no learning rate or optimizer for the IQL policies or value networks in its tables app. B.2.

Environment facts, app. C: AntMaze uses an 8-DoF ant. The observation is 29-dimensional. Goals are (x, y) positions. The reward is sparse, 1 at the goal and 0 elsewhere. D4RL provides "play" and "diverse" datasets, 1000 trajectories of 1000 steps each. The ultra map was introduced by another paper app. C §5.1. The extreme map has about 166% of the ultra surface area, and its datasets were collected like the smaller maps app. C.

Results

Author-reported. Setup: AntMaze medium, large and ultra, with D4RL "play" and "diverse" datasets, mean ± std over 8 seeds for QPHIL and HIQL §5.1 tab. 1. Baseline numbers for non-extreme mazes are copied from the HIQL paper app. A. Other baseline numbers come from their own papers §5.1.

Table 1: AntMaze success rate (%) tab. 1

Blank cells are empty in the paper. Bold marks the best value per row as typeset in the paper.

Extreme maze fig. 5: success rate (%), Diverse | Play.

Random-AntMaze (50 cycled start and goal pairs), success rate (%) app. D, random-AntMaze table:

Behaviour-cloning high-level policy (ultra only) app. D, BC table:

Re-planning, extreme only (success rate %) app. D, re-planning table:

The authors state that re-planning when the reached token differs from the planned one does not significantly change performance, and that 10 sampled plans, keeping the shortest that reaches the goal area, helps somewhat on extreme-play app. D.

Headline claims in the text: QPHIL is near state of the art on smaller mazes and reaches 70% success on AntMaze-Ultra §5.2. On AntMaze-Extreme it reaches up to 50% success, against 21.9% and 22.8% for HIQL without representation §5.3.

Paper vs code

No official code is used on this card. The paper's own appendix says the code is "available in the supplementary materials" and "will be next available in a public dedicated repository" app. B.1. The arXiv comments say "Code will be released upon acceptance". I checked the arXiv abstract page, then ran a GitHub repository search for "QPHIL" and "quantizing planner hierarchical implicit". It returned ubisoft/ubisoft-laforge-qphil, which matches the paper title and is under an owner that matches the authors' affiliation (Ubisoft La Forge). Its LICENSE is the "Ubisoft Machine Learning License (Research-Only - Copyleft)", which states that commercial use is strictly prohibited. Under the wikillm licensing rules, this repository cannot be cited, quoted or recorded in the code field, so this card contains no code citations and no code-derived checks. The paper text is the only source for the details above. I deleted the local clone after reading only its LICENSE file.

Pitfalls

• Seed count conflicts. Section 5.1 says QPHIL and HIQL ran 8 seeded runs §5.1. The reproducibility statement says "the same 8 seeds" but lists 7 values: 0, 1, 2, 3, 4, 6, 7 §8. Which seeds were used is ambiguous.
• Claim vs table, large-diverse. The text says QPHIL "outperforms all other methods on the larger maps" §5.2. In Table 1, HIQL w/ repr. scores 88.2 on large diverse, above both QPHIL variants (80.25 and 80.5). The claim does not hold for that row.
• IQL goal-policy tuple order. The text gives (β, τ, γ) = (0.99, 3, 0.9) for small mazes app. B.2. Read in that order, τ = 3, but τ must lie in [0.5, 1). The baseline grid (β ∈ {1, 3, 10}, τ ∈ {0.7, 0.9}, γ ∈ {0.99, 0.995}) app. A fits the order (γ, β, τ) = (0.99, 3, 0.9) and (0.995, 1, 0.7). This order is inferred from the grid, not stated.
• VQ-VAE coefficients unmapped. The table gives contrastive, commit, reconstruction and "norm" coefficients app. B.2, but the text writes the loss with α₁, α₂ and β without giving their values §4.1. The "norm coef" is never defined. The window δ is not given; "quantizer window 100" is the nearest candidate.
• Decoder target. The text defines reconstruction on the full state s §4.1. The appendix says the decoder decodes positions and that AntMaze inputs are noisy and normalized positions app. B.2. The implemented target is therefore likely positions, not states.
• Action dimension mismatch. The appendix writes the action space as [-1, 1]⁹ app. C, but the same section and the main text describe an 8-DoF ant §5.1.
• Stitching details. The compiled paper gives the splice rule in words §4.2 but no algorithm. The appendix's stitching pseudocode is commented out in the source, so it is not part of the compiled paper app. B.6. The number of augmented trajectories, sampling policy, and whether splices are added once or on the fly are not stated.
• Cleanup truncation. Algorithm 2 says to truncate "up to the first occurrence" of a repeated token alg. 2. It is unclear whether the repeated token is kept or dropped.
• Relabelling edge case. The next-landmark relabel is undefined when no later state has a different token, and the paper does not say what goal is used there §4.3.
• Goal sampling. The text says the next token is sampled with probability 0.8 and the "current" token with 0.2 app. B.2. It does not say what "current" refers to in the relabelling.
• Re-planning table does not match main results. The re-planning table's 1-sample numbers (35.5 play, 38.5 diverse) match neither the w/ aug. nor the w/o aug. numbers in Table 1 and the extreme table (50.0 / 39.5 w/ aug., 40.5 / 12.5 w/o aug.). The stitching setting used for re-planning is not stated.
• BC+IQL table reuses non-random numbers. The HIQL columns in the behaviour-cloning table (39.2, 56.0, 52.9, 52.6) are identical to the non-random ultra row of Table 1, though the table is labelled r-ultra. Only the BC+IQL columns are new.
• Naming. Table 1 calls the baseline "IQL" while the text and the extreme table say "GC-IQL". "w/ repr." and "w/o repr." are used throughout but not defined in this paper.
• Unequal baseline tuning. Non-extreme baseline numbers are copied from the HIQL paper app. A. Only HIQL, GC-IQL and HGCBC were re-tuned on extreme (grid over β, τ, γ, k). Comparisons on extreme use different tuning effort.
• Planner conditioning. The transformer takes the goal token and the full compressed history §4.2, but the exact input layout (special tokens, ordering) is not stated in the compiled paper.
• Sampling is stochastic. Plans are sampled at temperature 0.9 app. B.2, so results vary from run to run. Plan selection and the maximum step count of an episode are not stated.
• Validation split wording. The transformer table lists "validation split 0.95" app. B.2. The text says validation is computed with a "0.95 dataset split". Whether 0.95 is the training fraction is not stated.
• Unknown run count per setting. The paper gives about 10 GPU-hours per training run on V100 GPUs app. B.1. The number of runs in total, and the number of planner trainings per seed, is not stated.
• Cost. Training uses a VQ-VAE, a transformer and two IQL policies, and the stitching variant uses 250 planner epochs, against 2500 without stitching app. B.2. The paper reports both variants without a compute comparison.

Lineage

• Builds on: HIQL (the paper cites park2024hiql), which QPHIL replaces at the high level with discrete landmark planning §3 app. A.
• Uses: VQ-VAE for tokenization §4.1, with a triplet-margin contrastive loss (the paper cites balntas2016learning) §4.1. Transformers for planning §4.2. IQL and AWR for low-level policies §3 §4.3.
• Compared against: GCBC, IQL/GC-IQL, GC-POR, HIQL, HGCBC, Trajectory Transformer, Trajectory Autoencoding Planner, G-ADT and Planning Transformer §5.1.
• Superseded by: not known from content on this wiki.