SALON: Self-supervised Adaptive Learning for Off-road Navigation

In plain words

Driving off-road is hard to program by hand, because grass, gravel, bushes and rocks look different in every place. SALON lets a robot learn what is safe to drive on while it drives. It looks at the terrain with a general-purpose vision model, and it feels how bumpy each patch of ground is through its vehicle's motion sensors. It stores these pairs in a top-down map and predicts, for each patch, how rough it is and how fast the robot should go.

The robot does not need a long training run on each new terrain. A few seconds of experience is enough to start separating smooth ground from rough ground. A person can also click once on an object, such as a tree, to tell the robot that it must avoid that kind of thing. The authors say the result navigates about as well as systems that were trained on far more data, but that claim is theirs and has not been checked independently.

TL;DR

SALON is an online, self-supervised traversability learner for off-road robots. It builds a bird's-eye-view (BEV) map of visual foundation-model features, learns roughness from proprioceptive bandpower, and uses a Gaussian process with CVaR to output adaptive cost and speed maps §3. The authors report navigation comparable to offline-trained methods on a full-scale ATV with minimal human input §4.2. The official code is partial: the algorithm lives in a submodule that is pinned to an older commit (see Paper vs code).

Problem

Off-road robots must drive on terrain without markings or structure, and hand-made rules fail across varied scenes §1. Learned costmaps trained offline need large labelled or teleoperated datasets and can fail on domain shift §1, §2. Earlier proprioception-based adaptation works either need minutes of teleoperation per terrain type (WVN) or do not learn fast enough (TerraPN, about 25 minutes of learning) §2.2. SALON targets adaptation within seconds of experience with one hand-label, and it assumes that roughness rises roughly monotonically with speed §5.

Method

The pipeline has five parts. Each cycle, pixel features from a visual foundation model (VFM) are reduced and projected into a BEV map. Roughness is computed from vehicle signals and stored with the visual feature and the speed of the vehicle. A Gaussian process (GP) predicts roughness and speed for each cell. CVaR converts the prediction into a risk-aware value. Finally, a one-click label and an out-of-distribution test give lethal cells a high cost [§3, fig. 2].

1. Visual backbone. A VFM produces a feature image D with C channels per pixel from an RGB image §3.1.1 eq. 1. The authors chose VFMs for their generalisation to unseen terrain, and they accept lower spatial resolution §3.1.1.
2. Dimensionality reduction. Pixel features are sampled from training images. K-means finds k cluster centres. Each pixel is mapped to a k-dimensional descriptor whose k-th entry is the L1 distance to the k-th centre §3.1.2 eq. 2. The VLAD-style design follows a place-recognition method, and the authors say it keeps semantic properties §3.1.2.
3. BEV mapping. Visual features are matched to lidar points with known calibration and projected into a BEV grid. An exponential moving average over time smooths each cell §3.1.3. The mapping is the one used in Velociraptor §3.1.3.
4. Self-supervised roughness. Roughness is the weighted sum, over the acceleration axes a_x, a_y, a_z and suspension shock travel, of the bandpower of each signal over a frequency band §3.2. The weights, window lengths and bands are fitted so that roughness matches traversability scores that a passenger gave periodically on a 0 to 1 scale during a small data-collection drive §3.2.
5. Experience buffer. Each sample stores the BEV feature under the tire, the vehicle speed and the measured roughness §3.3. When the buffer is full, the method removes a sample from the most common semantic class and speed instead of the oldest one (FIFO). Table 1 reports average pairwise distance in the buffer: higher means better coverage §3.3 tab. 1.
6. Cost and speed prediction. A GP with an RBF kernel gives the mean and variance of roughness, given a cell's feature and speed §3.4 eq. 3. The authors prefer GP over an MLP because it adapts faster with stable predictions §3.4. The risk-adjusted roughness is the mean plus the variance times phi(Phi^-1(alpha_R)) / (1 - alpha_R), where alpha_R is the user's risk tolerance §3.4 eq. 4.
7. Speed map. The user sets a maximum roughness R_max between 0 and 1. The same GP predicts speed given the cell and R = R_max §3.4 eq. 5. A CVaR value with alpha_S, which adapts online, gives the speed limit §3.4. alpha_S rises while the robot is near the speed limit and the terrain is smoother than expected, and it falls if the terrain is rougher than R_max §3.4. The paper does not give step sizes.
8. One-shot cost augmentation. The user clicks one feature that belongs to a common lethal object, such as a tree. That feature is linked to high roughness in the buffer, which gives high cost to similar features without retraining §3.5. The paper does not state the roughness value used.
9. Out-of-distribution (OOD) avoidance. A cell is lethal if its distance to every cluster centre exceeds a threshold, using the uncertainty measure from Velociraptor. Morphological erosion and dilation then clean the result §3.5.1. The threshold value is not stated.

# SALON online update, one control cycle  [§3.1] [§3.2] [§3.3]
# Input:  image, lidar_points, imu_and_shock, speed
# State:  buffer of (feature, speed, roughness) samples
# Output: updated buffer

def update(image, lidar_points, imu_and_shock, speed, buffer):
    # Dense features from the visual foundation model  [§3.1.1] [eq. 1]
    features = vfm_features(image)                       # (H, W, C)
    # Pixel to k L1 distances from the cluster centres  [§3.1.2] [eq. 2]
    descriptors = l1_distance_to_centers(features, centers)  # (H, W, k)
    # Project into BEV with lidar calibration, smooth over time  [§3.1.3]
    bev_features = project_and_ema(descriptors, lidar_points)  # (X, Y, k)
    # Roughness from bandpower of the acceleration and shock signals  [§3.2]
    roughness = roughness_from_bandpower(imu_and_shock)
    # Store the feature under the tire cell with speed and roughness  [§3.3]
    sample = (bev_features[tire_cell], speed, roughness)
    buffer = maintain_buffer(buffer, sample)
    return buffer


# Keep the buffer at a fixed capacity  [§3.3]
def maintain_buffer(buffer, sample, capacity):
    # Fill until full; the capacity value is not stated in the paper
    if len(buffer) < capacity:
        return buffer + [sample]
    # Remove one sample of the most common (class, speed) bin  [§3.3]
    victim = most_common_bin_member(buffer, sample)
    return replace(buffer, victim, sample)

# Risk-adjusted value of a Gaussian prediction (CVaR)  [§3.4] [eq. 4]
# Used for roughness with alpha_R and for speed with alpha_S
def risk_adjusted(mean, variance, alpha):
    # alpha in (0, 1): the user's risk tolerance (roughness) or alpha_S (speed)
    quantile = inverse_normal_cdf(alpha)
    density = normal_pdf(quantile)
    return mean + variance * density / (1.0 - alpha)


# Speed limit for a cell at a roughness cap R_max  [§3.4] [eq. 5]
def speed_limit(buffer, cell_feature, r_max, alpha_speed):
    # GP posterior of speed given the cell feature and R = R_max  [§3.4]
    mean_speed, variance_speed = gp_posterior(buffer, cell_feature, r_max)
    # alpha_speed is adapted online  [§3.4]
    return risk_adjusted(mean_speed, variance_speed, alpha_speed)

# Lethal-object test from cluster distance  [§3.5.1]
def is_lethal(cell_feature, centers, threshold):
    # Distance measure is assumed to be the L1 distance of [eq. 2]; not stated
    distances = l1_distance_to_centers(cell_feature, centers)
    # Lethal when the cell is far from every cluster  [§3.5.1]
    return min(distances) > threshold   # threshold not stated

Key equations, copied in full with citations:

f_\theta(I_{3\times H\times W}) = D \in \mathbb{R}^{C\times H\times W} eq. 1

d_{VLAD}[k] = \| d - F_k \|_1 eq. 2

\mu_R, v_R = p(R \mid O, S) eq. 3

R = \mu_R + v_R \frac{\phi(\Phi^{-1}(\alpha_R))}{1-\alpha_R} eq. 4

\mu_S, v_S = p(S \mid O, R=R_{max}), \quad 0 \le R_{max} \le 1 eq. 5

The roughness of §3.2 is an unnumbered inline formula, shown here for completeness: R = \sum_i w_i\, BP(s_i, f^{min}_i, f^{max}_i) over i \in \{a_x, a_y, a_z, \text{shock}\} §3.2.

Hyperparameters

The paper states few numeric values. The table lists paper values first and then the base configuration in the official code, marked as code.

The base config in GP_base.yaml is one of several configs in the submodule. The paper does not say which config produced each result.

Results

Author-reported throughout. Setup: a full-scale Yamaha Viking ATV on a figure-8 course with five laps, a safety driver, and per-lap counts §4.2.3. Baselines run on the same platform: a geometric cost, GANav, HDIF and Velociraptor §4.2.1.

Navigation (figure-8, 5 laps) tab. 4. The table gives per-lap values. The totals and means below are sums or averages of those per-lap values, computed for this card and not stated in the paper.

The authors say their method is best on all metrics except average speed §4.2.3. Velociraptor is faster, and the authors accept that trade-off because they weight interventions more heavily §4.2.3.

Buffer sampling tab. 1. Average pairwise distance between buffer points, three scenarios:

Capability comparison tab. 3. The table marks each method's properties (self-supervised, online adaptation, risk-aware, OOD detection, velocity informed). SALON is marked with all five. The table is the authors' own assessment.

Qualitative results (no numbers): SALON separates tree line, trail and hidden objects in an ATV costmap within 60 s of experience §4.2.2 fig. 4; fine-grained gravel is high cost on a trail §4.2.2 fig. 5; grass is re-costed within a few seconds §4.2.2 fig. 6. On the ANYmal quadruped, the authors compare to WVN and say their costmap is more consistent §4.3 [fig. 7]. On the wheelchair, rough cobblestone is recognised as rougher than sidewalk within 5 s §4.4 fig. 8.

Headline claim: comparable navigation to methods trained on 100–1000x more data [abstract; §1]. The body does not show the dataset sizes behind this figure, so the claim is not checked here.

Paper vs code

Code: castacks/salon_ws (BSD-3-Clause), commit 5b65c3a9bc2d94a5e2fffa7c37d994b662e49302, linked from the paper's project page. The repository is a ROS1 workspace. Its README says "Documentation coming soon" and that the wheelchair data "will be available soon" [code: README.md]. The algorithm sits in two git submodules. This card covers castacks/context_adaptation at the pinned commit f697cd72c63416f10b83fbb8cd1e287f577f96ef. The other submodule, physics_atv_visual_mapping, was not read. The main branch of context_adaptation has a different file layout (for example salon.py), so the code the paper refers to is not unambiguous.

The script src/context_adaptation/scripts/dino_costmap_gp_speed_input_output.py at the pinned commit was read only through a tool-generated summary, not as the raw file. Its line numbers are not verified, so the citations below omit them. Re-check every item against a clone before implementing. The base config citations are line-numbered, but they come from the same kind of rendering.

Differences between paper and code:

• GP online training. The paper describes GP regression with an RBF kernel and no online fitting §3.4. The code refits the GP every 10 cycles, and it fine-tunes the speed GP with SGD at learning rate 0.01; the cost-GP training block is commented out dino_costmap_gp_speed_input_output.py.
• Speed GP inputs. The paper conditions speed on the feature and on R = R_max eq. 5. The code uses a subset of feature columns for the speed GP, indices 0 to 7 and 9 dino_costmap_gp_speed_input_output.py.
• One-shot label. The paper links one clicked feature to high roughness §3.5. The code seeds 8 samples of one class with label 1.0 and velocities spread by 1.2, and it never evicts those 8 dino_costmap_gp_speed_input_output.py.
• Buffer writes and eviction. The paper stores experience as the robot drives and removes a sample from the most common class and speed §3.3. The code writes only when speed is above 0.3 and every buffer_update_freq cycles GP_base.yaml dino_costmap_gp_speed_input_output.py. It evicts a random sample from the most common class and one of 10 velocity bins over 0 to 10 dino_costmap_gp_speed_input_output.py.
• Costmap scaling. The code divides the costmap by 2.0 before publishing dino_costmap_gp_speed_input_output.py. The paper does not mention a scale.
• Uncertainty override. Where the uncertainty map is nonzero, the code writes its values over the cost map dino_costmap_gp_speed_input_output.py. The paper says a lethal cell is one far from all clusters, followed by erosion and dilation §3.5.1. The code's erosion and dilation were not confirmed.
• Alpha_S rules. The paper says alpha_S is increased when the robot is near the limit and terrain is smoother, and decreased when rougher §3.4. The code adds +0.005, −0.02, and an extra +0.01 when the speed error exceeds 0.4 and the predicted speed is below 2.8, clipped to [0, 0.99]. It also resets alpha to 0 and sets the speedmap to 4.5 during warm-up dino_costmap_gp_speed_input_output.py.
• Warm-up. Until the buffer holds 5 + 8 samples, the code outputs zero maps or the default 4.5 speed; the paper is silent dino_costmap_gp_speed_input_output.py.
• Experiments. The paper uses DINOv2 on the ATV §4.1.1 and RADIOv2.5 on the wheelchair §4.1.2. The repository's launch files cover the three robots README.md. The README says the wheelchair data is not yet available, so the wheelchair result cannot be reproduced from the repository.

Checked and matching: the CVaR formula in the code has the same form as eq. 4. Not checked: the VLAD distance, the bandpower roughness and the MPPI controller.

Pitfalls

• Partial code. salon_ws contains only the workspace, the README and a launch-file list. The algorithm is in a submodule that is pinned to an older commit. Do not implement from the top-level repository alone.
• Licence of the submodule. The pinned context_adaptation commit has no LICENSE file. The main branch lists one, but its text was not retrieved. The code_license field covers only castacks/salon_ws. Do not copy code from the submodule until its licence is checked.
• Unstated numbers. Buffer size, cluster count, bandpower parameters, the one-shot roughness value, the OOD threshold, the BEV EMA rate and all MPPI settings are missing from the paper. The code's base values are not the paper's experiment values. Check which config produced each result.
• Units and scales. Annotations run from 0 to 1 §3.2, but the base config uses max_cost 0.25 and the code divides the costmap by 2. The maximum speed of 6 has no stated unit.
• Sensitive constants. alpha_S step sizes, the 2.8 speed check and the 0.4 error threshold are code-only and tuned by the authors. They shape how fast the robot explores.
• Monotonicity assumption. The speedmap assumes roughness rises with speed. The authors state that some terrain may be smoother at speed §5.
• Evaluation scope. Table 4 covers one figure-8 course with five laps, one ATV, one safety driver, and no variance. Interactions and undesirable behaviour are counts judged by the driver. Totals in this card are derived from per-lap values and are not the authors' numbers.
• Tie on interventions. At R_max = 0.4 SALON has 4 interventions, the same total as Velociraptor, so "best in all metrics" is not literally true for that setting.
• Wheelchair and ANYmal. These are qualitative only §4.3 §4.4.
• Exact GP. The code uses an exact GP. The paper gives no runtime or memory figures for the buffer size.

Lineage

• Builds on: Velociraptor (BEV mapping, uncertainty-based OOD test) §3.1.3 §3.5.1; HDIF and Castro et al. (bandpower roughness) §3.2; WVN (ANYmal data and velocity-error cost) §4.1.3; TartanDrive (ATV platform) §4.1.1. None of these has a card in this wiki yet.
• Compared against: a geometric cost from ALTER, GANav, HDIF, Velociraptor §4.2.1; WVN qualitatively §4.3.
• Superseded by: not known from this wiki.