---
kind: concept
slug: visual-inertial-odometry
title: Visual-inertial odometry
summary: >-
  Visual-inertial odometry estimates a platform's pose by fusing camera measurements with IMU readings. Kimera's VIO runs on a camera and IMU alone. Three cards use it inside lidar-inertial-visual systems: R2LIVE, LVI-SAM and R3LIVE. Accuracy is author-reported.
tags: [state-estimation, sensor-fusion, visual, slam]
papers: ["2101.06894", "2102.12400", "2104.10831", "2109.07982"]
related: [error-state-kalman-filter, lidar-inertial-odometry, simultaneous-localization-and-mapping, pose-graph-optimization, 3d-scene-graph]
status: draft
updated_at: 2026-10-31
---

## Overview

Visual-inertial odometry (VIO) tracks a sensor rig's pose by combining an
inertial measurement unit (IMU), which reports acceleration and rotation at a
high rate, with a camera, which tracks image features at a lower rate. The IMU
predicts the motion between camera frames, and the camera's feature
measurements constrain that prediction [arxiv:2101.06894 §3.1]
[arxiv:2102.12400 §IV-D, §IV-H] [arxiv:2104.10831 §2.2, §2.3].

Four papers with cards cover this. Kimera's VIO module needs only a camera and
an IMU, with stereo or RGB-D images [arxiv:2101.06894 §3]. The other three put
the camera in a lidar-inertial-visual system, where the visual-inertial part
works next to a lidar-based part. R2LIVE adds the camera to a lidar-inertial
filter [arxiv:2102.12400 §I]. LVI-SAM runs a visual-inertial sub-system and a
lidar-inertial sub-system side by side, and each gives the other a starting
guess [arxiv:2104.10831 §2.1]. R3LIVE runs a visual-inertial step against a
map that its lidar-inertial part builds [arxiv:2109.07982 §II]. Cards:
[Kimera](/papers/2101.06894), [R2LIVE](/papers/2102.12400),
[LVI-SAM](/papers/2104.10831) and [R3LIVE](/papers/2109.07982).

## Variants

Five variants. The first comes from Kimera and runs on a camera and IMU alone.
The next two come from R2LIVE and differ in where the camera measurements enter.
The fourth comes from LVI-SAM, where the visual-inertial part gets its depth from
the lidar. The fifth comes from R3LIVE, which updates the state against a
lidar-built map using photometric residuals.

### Keyframe fixed-lag smoother, camera and IMU only

Kimera-VIO estimates pose in a keyframe-based maximum a posteriori problem with
IMU preintegration [arxiv:2101.06894 §3.1]. Shi-Tomasi corners are tracked with
Lucas-Kanade at every frame. The tracker is seeded with rotation predicted by the
IMU, because the robot moves through crowds [arxiv:2101.06894 §3.1]. Feature
detection, stereo matching and geometric verification run only at keyframes
[arxiv:2101.06894 §3.1]. The back end is a factor graph solved with iSAM2 in
GTSAM. Each feature's 3-D point is triangulated and then eliminated from the
state, so the graph has no landmark variables [arxiv:2101.06894 §3.1]. The
smoothing horizon is fixed-lag, with 6 s as the default in the TX2 configurations
[arxiv:2101.06894 §4.9]. Its verification defaults are IMU-aided RANSAC, with
2-point for stereo and 1-point for mono [arxiv:2101.06894 §3.1].

### Camera update inside an error-state filter

Camera measurements are fused at each camera time, in the same iterated filter
that fuses LiDAR and IMU [arxiv:2102.12400 §IV-E, §IV-F]. Each tracked corner
gives one reprojection residual, the difference between the observed pixel and
the projection of a 3D landmark through the IMU and camera extrinsic
[arxiv:2102.12400 §IV-H, eq. 16]. Corners are found with FAST and tracked with
KLT optical flow [arxiv:2102.12400 §IV-H]. The measurement noise enters as the
pixel noise covariance, which the paper does not give a value for
[arxiv:2102.12400 §IV-H, eq. 20].

### Sliding-window visual factor graph

A back end optimizes camera poses and landmark positions over a sliding window
of image keyframes [arxiv:2102.12400 §V]. It adds IMU preintegration factors
between keyframes and estimates the camera-LiDAR time offset. The authors say
the graph is similar in structure to VINS-Mono's [arxiv:2102.12400 §V]. It runs
in a separate thread at a lower rate than the filter [arxiv:2102.12400 §VI-F].

### Visual-inertial sub-system with lidar depth

LVI-SAM's visual-inertial sub-system adapts the VINS-Mono pipeline. It tracks
corners with KLT and optimizes a sliding window per frame. The state holds
rotation, position, velocity and IMU biases [arxiv:2104.10831 §2.2]. Lidar depth
is attached to each feature when it is available. The depth comes from lidar
frames registered to the camera with the visual odometry, and a feature keeps it
only if the three nearest depth points lie within 2 m of one another
[arxiv:2104.10831 §2.2.2]. In the Urban ablation, the visual-inertial run
without lidar depth has a translation error of 239.19 m, against 142.12 m with
depth [arxiv:2104.10831 tab. 1]. Loop candidates are found with DBoW2 in this
part and checked by the lidar-inertial part [arxiv:2104.10831 §2.2.4, §2.3]. The
visual-inertial pose is passed to the lidar-inertial sub-system as the starting
guess for scan matching, and IMU readings with corrected bias fill in when it is
not available [arxiv:2104.10831 §2.3.1].

### Photometric update that colours a lidar map

R3LIVE's visual-inertial step runs after its lidar-inertial part has built a
global point map. Each image updates the state twice, both times with the same
iterated error-state filter [arxiv:2109.07982 §V-A2, §V-B2]. The first update
uses frame-to-frame reprojection residuals on points tracked with Lucas-Kanade
[arxiv:2109.07982 §V-A1]. The second uses frame-to-map photometric residuals: the
colour stored in a map point, minus the colour seen at the point's predicted
pixel [arxiv:2109.07982 §V-B1]. The observed colours are then fused into each
map point by a precision-weighted average [arxiv:2109.07982 §V-C]. The state also
holds the camera intrinsics and the camera-IMU time offset, estimated online
[arxiv:2109.07982 §III-A]. The paper states that the method needs no extracted
visual features, and that camera-based methods struggle on texture-less surfaces
[arxiv:2109.07982 §I].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Camera and IMU only, stereo or RGB-D images, no lidar | Keyframe fixed-lag smoother (Kimera-VIO) | On the paper's EuRoC table, Kimera-VIO has the lowest error of the fixed-lag methods on 10 of 11 sequences; on V2_03, ROVIO is lower [arxiv:2101.06894 tab. 1]. Timings are on an unnamed platform: feature tracking takes 7.5 ms per frame and keyframe processing 51 ms [arxiv:2101.06894 §4.8] |
| LiDAR, camera and IMU available, and the camera may be blocked or dark | Camera update in the filter, with the LiDAR and IMU carrying the estimate | With the camera covered, the paper reports a smooth trajectory [arxiv:2102.12400 §VI-B, fig. 7] |
| Fast rotation, up to 300°/s in the authors' test | Keep the filter at IMU rate, with camera and LiDAR updates | The paper reports usable tracking in this test [arxiv:2102.12400 §VI-B] |
| Lidar-based odometry is expected to fail on some sequences | Visual-inertial sub-system with lidar depth, plus the lidar-inertial sub-system | On the Handheld dataset, LOAM, LIO-mapping and LINS fail, and LVI-SAM reports 0.83 m RMSE with loop closure [arxiv:2104.10831 tab. 2] |
| Long tunnel with pedestrians, where visual tracking may fail | Visual-inertial path with the LiDAR as backup | In the authors' run, VINS-Mono stopped midway on feature-tracking failure, and the paper reports R2LIVE's trajectory as the best of the three methods [arxiv:2102.12400 §VI-C, fig. 9] |
| Route that returns to a visited place, where drift matters | Loop closure with visual candidates, verified against the lidar map | On the Jackal dataset, LVI-SAM's RMSE is 0.67 m with loop closure and 4.05 m without [arxiv:2104.10831 tab. 2] |
| Texture-less walls, a LiDAR map is wanted, and the output needs colour per map point | Photometric update against the lidar map (R3LIVE) | In the authors' texture-less passage with bare white walls, the end-of-trajectory drift is 1.62° rotation and 4.57 cm translation [arxiv:2109.07982 §VI-B, fig. 8]. Cost is 7.01 to 43.71 ms per frame on the desktop and 15.00 to 166.08 ms onboard, rising with image size and finer point spacing [arxiv:2109.07982 tab. IV] |
| Onboard compute of about 8 GB and a low-power CPU | Filter for real-time, factor graph as a slower stage | Author-reported: the visual filter step takes 13.92 ms onboard in Exp-1, and the factor graph 45.35 ms [arxiv:2102.12400 tab. I] |

Accuracy numbers are author-reported. On a 300 m sequence against D-GPS, the
median translation error is 3.84% for VINS-Mono, 0.34% for FAST-LIO and 0.21%
for R2LIVE [arxiv:2102.12400 fig. 11]. LVI-SAM reports RMSE against GPS on its
own Jackal and Handheld datasets [arxiv:2104.10831 tab. 2]. Kimera-VIO's ATE on
EuRoC ranges from 0.05 m to 0.21 m [arxiv:2101.06894 tab. 1]. The paper aligns
trajectories with SE(3), not Sim(3), and says Sim(3) would give a smaller error
for Kimera [arxiv:2101.06894 §4.2]. R3LIVE reports relative pose error against
RTK ground truth in a seaport, for example 0.21 deg and 0.17% at 300 m for
R3LIVE-HiRes [arxiv:2109.07982 tab. III]. Each paper's test uses its own dataset
and metric, so do not rank these methods against each other from these numbers.
LVI-SAM's timing is not measured in the paper. Its shipped configuration caps the
visual solver at 0.035 s and 10 iterations [code: config/params_camera.yaml#L82-L83],
which are limits, not measured figures.

## Implementation sketch

The sketch covers the camera side, which is specific to VIO, and the handoff
from the visual-inertial part to the lidar-inertial part in LVI-SAM. The first
two blocks follow R2LIVE, and the third follows R3LIVE's photometric update.
The IMU propagation and the iterated update are described in the
[error-state Kalman filter](/concepts/error-state-kalman-filter) concept page.

```pseudocode
# Track corners and measure their pixels at each camera time  [§IV-H]
def camera_measurements(prev_frame, frame, tracked_corners):
    # Carry corners forward with KLT optical flow, detect new FAST corners
    # [§IV-H]
    corners = klt_track(prev_frame, frame, tracked_corners)
    corners += fast_detect(frame, exclude=corners)
    return corners
```

```pseudocode
# One reprojection residual per corner, for the iterated update  [§IV-H] [eq. 16]
def camera_residuals(state, corners, landmarks):
    residuals = []
    for pixel, landmark in zip(corners, landmarks):
        # Landmark in camera frame through IMU and camera extrinsic  [eq. 16]
        point_cam = world_to_camera(state, landmark)
        # Observed pixel minus the projection of the landmark  [§IV-H]
        residuals.append(pixel - pinhole(point_cam))
    return residuals
```

```pseudocode
# Photometric residual of one lidar map point in the current image  [§V-B1]
def photometric_residual(state, point, image):
    # Predicted pixel, with the estimated time-offset correction  [§V-A1]
    pixel = project_with_time_offset(point, state)
    # Colour at that pixel, interpolated from neighbouring pixels  [§V-B1]
    observed = bilinear_colour(image, pixel)
    # Stored map colour minus observed colour  [§V-B1]
    return point.colour - observed
```

```pseudocode
# Visual-inertial pose seeds the lidar scan match  [§2.3.1]
def lidar_initial_guess(vis_pose, vis_valid, imu_readings, bias):
    # Use the visual-inertial pose while it is available  [§2.3.1]
    if vis_valid:
        return vis_pose
    # Otherwise integrate IMU readings with the corrected bias  [§2.3.1]
    return integrate_imu(imu_readings, bias)
```

Gating of camera updates is not described in the paper. The card's
[Paper vs code](/papers/2102.12400) section notes that the authors' code applies
acceptance gates, which an implementation must decide for itself.

## Contradictions

Within R2LIVE, the paper and its code differ on the camera covariance update and
on gating. See the card's [Paper vs code](/papers/2102.12400) section.

Within LVI-SAM, the paper and its code differ in four places. The paper says the
lidar-odometry factor is dropped when degeneracy is detected, but the code always
adds it and only projects out the degenerate directions [arxiv:2104.10831 §2.3.2]
[code: src/lidar_odometry/mapOptmization.cpp#L1377-L1381]. The paper lists
visual-odometry factors in the lidar-inertial graph, but the code comments that
factor out [arxiv:2104.10831 §2.3]. The paper includes a feature-count failure
trigger for the visual part, but its check is disabled in the code
[arxiv:2104.10831 §2.2.3]. See the card's
[Paper vs code](/papers/2104.10831) section for the citations.

The LVI-SAM text and its ablation table disagree on rotation. The text says depth
reduces error, while the rotation error for the depth-enabled A3 run rises from
6.41° to 7.66° [arxiv:2104.10831 §3.1.3, tab. 1].

Within R3LIVE, the colour-fusion equation printed in the paper, read literally,
takes the inverse of the weighted sum before multiplying by the covariance. The
code, and the card, use the precision-weighted form instead [arxiv:2109.07982
§V-C]. The paper and its code also differ in the VIO. The paper's cost is a plain
squared Mahalanobis term, while the code applies a Huber scale to every residual
in both updates [arxiv:2109.07982 §V-A2, §V-B2]. The paper describes removing
points with large residuals [arxiv:2109.07982 §V-D], but the code's residual
rejection is skipped for the argument it passes. The code also runs a
fundamental-matrix RANSAC that the paper does not mention. The paper gives no
iteration count for the photometric update, and the code runs a fixed two
iterations [arxiv:2109.07982 §V-B2]. See the card's
[Paper vs code](/papers/2109.07982) section for the code paths. The paper does
not say whether the VIO can run without the lidar map, so a camera-only port needs
its own depth source [arxiv:2109.07982 §II, §V].

## Papers

- [Kimera](/papers/2101.06894): a camera-and-IMU VIO module with a keyframe fixed-lag smoother, which the paper reports as competitive on EuRoC; the wider system builds a 3D scene graph.
- [R2LIVE](/papers/2102.12400): a LiDAR-inertial-visual filter with a visual factor graph in the back end, reported as robust to a covered camera and to fast rotation.
- [LVI-SAM](/papers/2104.10831): a lidar-visual-inertial factor graph with a visual-inertial sub-system that uses lidar depth, reported with the lowest RMSE against GPS on its Jackal and Handheld datasets, with loop closure.
- [R3LIVE](/papers/2109.07982): a LiDAR-inertial-visual system whose visual part colours a lidar map by photometric residuals, reported with translation drift of 0.09 to 0.16 m over 1.2 to 1.5 km campus runs.