Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving

In plain words

A vehicle on rough ground does not always respond the same way. Mud, sand, grass and slopes change how much the wheels grip and slide, so a planner needs a model that expects these changes. This paper adds the camera view of the ground to such a model.

The model is a physics-style vehicle model with a learned correction on top. The ground features come from a large pretrained vision model that already describes surfaces well. Those features are large, so a small learned encoder squeezes them into a few numbers per wheel. The encoder is trained together with the motion model, so it keeps the parts that help predict motion and drops the rest, such as lighting.

The authors tested the approach on a large set of aggressive off-road driving. With features taken from a viewpoint that is only possible after the fact, the model is about 10% better than the same model without vision. At realistic viewpoints the gain only holds for terrain up to about 30 metres ahead.

TL;DR

• A hybrid model: a bicycle-type physics model for the forces, plus an LSTM that corrects them §4, §4.1.
• Terrain input is DINOv2 ViT-S/14 features, reduced by PCA to 40 dimensions, mapped onto a 2D terrain grid, and compressed per wheel by an encoder trained end-to-end §3.2, §5.
• Training draws one projection distance per trajectory from hindsight or seven distance buckets, so the model works where features are collected at varying distances §5.1.
• Author-reported: about 10% lower mean summed loss than the no-feature baseline on hindsight features, and a naive model fails at larger distances §6.1, §6.2, §7.
• No official code was found. The results are plots, and the paper gives no runtime or memory figures.

Problem

High-speed driving on unstructured terrain changes the vehicle's dynamics suddenly, through traction, tyre slip and rolling resistance §1, abstract. A dynamics model that is only fitted to the vehicle does not anticipate these changes. Earlier terrain work mostly used visual information for traversability costs, not for dynamics §2. The closest DINO-based work uses terrain features to set two physical parameters, stiffness and friction; the other uses per-frame features for adaptive control §2.

The authors report a dataset of hundreds of kilometres of driving (abstract), with about 2 million 5-second trajectories for training §6.

Assumptions and limits that matter for implementation:

• The map is local and robot-centric. A feature is used only where a LiDAR return exists or a hole is filled from a neighbour within 0.4 m §3.2.3.
• At runtime, the feature for each cell comes from the encoder. In the dataset, it comes from PCA §3.2.3, fig. 1.
• The hindsight features come from a map that stores the last valid data, including rear LiDAR and camera data behind the vehicle. Runtime does not have these §3.2.4, §6.2.
• Occlusion and invalid features grow with distance. Beyond about 30 m the features are unreliable, and the paper shows no gain there §5.1, §6.2.
• The paper does not test new terrain types; this is named as future work §7.

Method

Mapping visual features §3.2. Four RGB cameras face forward, back and to both sides. The front and side cameras run at 10 Hz and the rear camera at 2 Hz, all at 960×594 pixels §3.2.1. Images are rectified and passed through DINOv2 ViT-S/14, which gives one 384-dimensional vector per 14×14 patch §3.2.1, fig. 3. PCA maps these to 40 dimensions. The PCA basis is fitted on ground pixels from 175 hand-picked images §3.2.2. LiDAR points are projected into the feature image and accumulated into a 3D voxel map with 0.2 m cells. Each cell keeps the closest observation §3.2.3. The 2D map takes the lowest valid voxel in each column, which the authors assume is the ground §3.2.3. Gaps are filled from the nearest cell within 0.4 m §3.2.3.

# Visual feature map  [§3.2.1] [§3.2.2] [§3.2.3] [fig. 3]
# Input:  four RGB images, LiDAR points, camera projections
# Output: 2D terrain feature map (one feature vector per cell)

def build_feature_map(images, lidar_points):
    # Rectify each image, then run the DINOv2 ViT-S/14 encoder  [§3.2.1]
    tokens = vfm_features(rectify(images))  # (68, 42, 384) per image
    # Reduce every token to 40 values with the fitted PCA  [§3.2.2]
    reduced = pca_project(tokens, basis)    # (68, 42, 40)
    # Project LiDAR points onto the image and attach the feature vector
    # [§3.2.3]
    points = attach_features(lidar_points, reduced)
    # Keep the closest observation in each 0.2 m voxel  [§3.2.3]
    voxels = closest_per_voxel(points, cell_size=0.2)
    # Keep the lowest valid voxel in each vertical column  [§3.2.3]
    grid = lowest_valid_per_column(voxels)
    # Fill holes from the nearest cell within 0.4 m  [§3.2.3]
    return fill_gaps(grid, radius=0.4)

Hybrid vehicle model §4. The terradynamics model predicts the state x = [p_x, p_y, φ, v_x, v_y, r] (position, yaw, body velocity, yaw rate) §4. Brake pressure, steering position and velocity, and engine RPM are predicted by separate delay models that are held fixed while the terradynamics is trained §4. The engine RPM is predicted directly from throttle and speed §4. The parametric bicycle model gives a force vector F eq. 1. Two modified slip angles are used eq. 2, eq. 3. The forces are converted into body-rate derivatives and integrated by forward Euler with Δt = 0.02 s eq. 4 §4.1. An LSTM ζ_μ predicts a compensation to the force vector from a 0.2 s history window eq. 5 §4.

\mathbf{F} = \begin{bmatrix} \left(P(x_{rpm})\,P(u_{th}) - P(x_{br}) - \beta(v_x)\right)\eta_z \\ D_R\sin\!\left(C_R\tanh(B_R\alpha_R)\right)\eta_z \\ D_F\sin\!\left(C_F\tanh(B_F\alpha_F)\right)\eta_z \\ \dfrac{v_x}{C_L}\,\delta\,C_r - C_{r,d}\,r \end{bmatrix} eq. 1

\alpha_R = \arctan\!\left(\frac{v_y - L_R\,r}{\max(C_{max},\,v_x)}\right) eq. 2

\alpha_F = \arctan\!\left(\frac{v_y + L_F\,r}{\max(C_{max},\,v_x)}\right) - \delta eq. 3

\dot v_x = \frac{(1+\cos\delta)F_x - F_{yf}\sin\delta}{m} - C_{x,d}v_x^2 - C_{x,g}\eta_x + v_y r

\dot v_y = \frac{F_{yb} + \cos\delta\,F_{yf} + F_x\sin\delta}{m} - C_{y,d}v_y^2 - C_{y,g}\eta_y - v_x r

\dot r = F_r, \qquad \dot{\mathbf p} = R(\phi)\,\mathbf v, \qquad \dot\phi = r eq. 4

\dot{\hat{\mathbf x}}_t = h\!\left(\mathbf F_t + \zeta_\mu(\hat{\mathbf x}_t, \mathbf u_t, \mathbf y_t, \mathbf F_t),\ \hat{\mathbf x}_t\right) eq. 5

The paper's baseline uses equation 5 §4. Its visual model adds the encoder output as an extra input to the LSTM:

\dot{\hat{\mathbf x}}_t = h\!\left(\mathbf F_t + \zeta_\mu(\hat{\mathbf x}_t, \mathbf u_t, \mathbf y_t, \mathbf F_t, \zeta_E(\hat{\mathbf y}_f)),\ \hat{\mathbf x}_t\right) eq. 6 §5.1

Feature encoder §5. A fully connected network ζ_E maps the 40 PCA values of one wheel location to n_encoder values. It has hidden layers of 64 and 32 units with tanh activations §5. The four wheels are encoded separately, so the LSTM receives 4·n_encoder extra inputs §5. The encoder is trained with the dynamics model end to end fig. 1.

Distance-independent training §5.1. In the map, the projection distance varies with the distance from the vehicle to the map origin, so each trajectory has several feature sets §3.2.4. The paper draws one feature set per trajectory from hindsight or one of seven distance buckets, 10 m apart, and keeps it fixed for the whole trajectory §5.1. Redrawing the set each step performed poorly §5.1. The sampling distribution is not stated.

# Distance-independent feature choice for one trajectory  [§5.1]
# Input:  feature sets for hindsight and seven distance buckets
# Output: one feature set, held fixed for the whole trajectory

def choose_feature_set(feature_sets):
    # Draw once per trajectory, not per step. The sampling distribution is
    # not stated; uniform is an assumption  [§5.1]
    index = random_index(len(feature_sets))
    return feature_sets[index]

# Hybrid rollout over one 5 s horizon  [§4] [§5.1] [eq. 1-6]
# Input:  initial state, controls, wheel feature vectors, dt = 0.02 s
# Output: predicted states over the horizon (T, 6)

def rollout(state, controls, wheel_features, dt):
    # Start from the measured state  [§4]
    predicted = [state]
    for t in range(T):
        # Forces from the bicycle model  [§4.1] [eq. 1-3]
        forces = bicycle_forces(predicted[t], controls[t])
        # Encode each wheel's features, one encoder run per wheel  [§5]
        # [eq. 6]
        codes = [encoder(f) for f in wheel_features[t]]
        # The LSTM sees its own predicted state and the actual inputs
        # over the last 0.2 s  [§4] [eq. 6]
        correction = lstm(predicted[t], controls[t], forces, codes)
        # Body-rate update, forward Euler  [§4.1] [eq. 4]
        rate = body_rates(forces + correction, predicted[t])
        predicted.append(predicted[t] + dt * rate)
    return predicted

Hyperparameters

Results

Author-reported. Setup: a large driving dataset with a no-feature baseline B. Hardware, sample counts and per-model numbers are not stated in the text. The per-model distance errors appear only as box plots, so the exact values are not given here fig. 4 fig. 5.

Other reported findings:

• The compressed model is slightly worse than feeding raw features directly (DF) §6.1. Feeding raw features is not feasible at runtime §6.1.
• Varying the encoder size or the PCA dimension gives similar performance, which the authors read as no loss of dynamics information §6.1 fig. 4b, fig. 4c.
• The paper gives no runtime, latency or memory measurements. The claims of a lightweight map and no significant computational burden are qualitative §5.1 §7.

Paper vs code

No official code. The paper has no code link: the abstract links only a YouTube video [abstract]. No project page is linked. A GitHub search for "dinov2 terrain dynamics", "visual terrain features dynamics", "mppi dinov2" and within the ACDSLab organisation, which the authors' lab uses, found no repository for this paper. The ACDSLab repository "ACDS-Dynamics-Model-Learning" (MIT) is a lab repository, but its README cites arXiv 2605.12628 and does not mention this paper or DINOv2, so it is not treated as this paper's code.

Pitfalls

• Equation numbers. The numbers above follow the order of the LaTeX source. The bundled PDF is the IEEE template sample, not the paper, so verify the numbers against the arXiv PDF before relying on them.
• Undefined symbols. η_x and η_y appear in equation 4, but the text defines only η_z. ŷ_f in equation 6 is never defined; it is presumably the feature set chosen for the trajectory, but the text does not say so.
• Equation 4 as written. The first row has (1+cos δ)·F_x, which the text does not explain. Check it before implementing. No code is available to confirm it.
• Tyre model naming. The text calls the tyre model Pacejka, but gives only the form sin(C·tanh(B·α)) and no Pacejka constants; the fitted constants are not stated. The max(C_max, v_x) denominator in equations 2 and 3 is the only motivation given, and C_max sets the trade-off between slip-angle accuracy and stiffness §4.1.
• Engine and delay models. The engine RPM is predicted from throttle and speed, but the function is not given §4. The brake and steering delay models are not described beyond their states §4.
• LSTM sizes. "Initialization LSTMs have 20 hidden layers" §6 may mean 20 units. The predictor's hidden size is given as 4, which is very small. Both are ambiguous. The LSTM initialisation follows the authors' earlier work [multistep], which is not in this paper.
• Distance range. The text says the seven buckets go up to 40 m ahead. The occlusion list also includes −20 m and −10 m, which are behind the vehicle §5.1. Negative distances use the back LiDAR and camera. The range the model is trained on is therefore ambiguous.
• Occlusion values. The list [1, 0.87, 0.92, 0.92, 0.84, 0.65, 0.42, 0.21] is labelled as occlusion percentages, but the values are fractions with hindsight = 1 §5.1. Read them as the share of valid data.
• Missing features. The flag is {−1, 1} with the training mean as the fill value §5.1. The text does not say whether the flag is per wheel or per whole feature set.
• Loss and optimiser. The "mean summed loss" is used for all comparisons but is not defined. The learning rate, batch size and loss weights are not stated.
• Split. The text says the training and test sets come from the same logs without overlap §6. How the split was made is not stated, so leakage between train and test cannot be ruled out.
• Hindsight is not runtime. Hindsight features use the last valid data, including data from the rear sensors §3.2.4. The headline 10% gain is for these features. The realistic result is the distance-independent one §6.2.
• Rear camera rate. The rear camera runs at 2 Hz against 10 Hz for the front and side cameras §3.2.1. Its features are therefore older, and the paper does not report how this affects the result.
• Narrow data. Most data (64%) comes from one site §6. No test on new terrain types is reported §7.
• Map choices. The 0.2 m voxels, closest-observation fusion, lowest-voxel rule and 0.4 m hole-fill radius are design choices of the RACER stack §3.2.3. The paper gives no ablation for them.
• Cost not in this paper. The MPPI cost and variant are described in [colored_mppi], which is not part of this paper §3.1.

Lineage

• Builds on: the authors' earlier multistep hybrid dynamics work [multistep]; the bicycle model form from [kabzan]; the MPPI planner [mppi] and its GPU library [mppi-generic]; and DINOv2 [dino]. None is compiled here, and their arXiv ids are not given in the source.
• Compared against: the no-feature baseline B, and the DINO-based work [hutter] and [magic_vfm] discussed in §2.
• Related: traversability-map work that uses visual terrain features, such as [bevnet] and [evora] §2.