wikillm

Paper · 2019-03-04 · cs.CV, cs.AI, cs.LG, cs.RO

Selective Sensor Fusion for Neural Visual-Inertial Odometry

Monocular visual-inertial odometry with learned masks that re-weight concatenated visual and IMU features. Soft (sigmoid) and hard (Gumbel-softmax) masks beat direct concatenation on corrupted KITTI, EuRoC and PennCOSYVIO data, with smaller gains on clean data. Results are author-reported.

Visual-inertial odometryMultimodal fusionsensor-fusionrobustnessodometry

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A camera and an inertial sensor (accelerometer and gyroscope) can both tell a moving device how it has moved. The camera is good at direction and turning but cannot measure distance travelled at true scale. The inertial sensor is fast and works in the dark, but it drifts over time. Real sensors also fail: frames get dropped, lenses get dusty, and clocks disagree. Many learned systems simply stack the two streams together and hope for the best, so a bad camera feed can drag the estimate off course.

This paper adds a gate between the two streams. A small network looks at both sets of features and decides, for each feature, how much to let through. The gate is trained together with the rest of the network. The authors try two kinds of gate: a smooth one that scales each feature between zero and one, and a stricter one that either passes a feature or blocks it. Both are compared with plain concatenation on three public datasets, including corrupted versions of them. The gates help most when the input is damaged. Looking at the gate values also shows which sensor the network trusts in a given situation, for example the inertial stream during fast turns.

TL;DR

  • Monocular visual-inertial odometry (VIO) with a learned fusion step that re-weights visual and IMU features before a recurrent pose regressor § 2 § 3.
  • Soft fusion uses sigmoid masks; hard fusion samples binary masks with Gumbel-softmax so it trains end to end § 3.2 § 3.3.
  • On corrupted data the gated variants beat direct concatenation on most columns, while clean-data gains are small or absent tab. 2 tab. 3 tab. 4.
  • The released code trains only the hard variant, freezes the visual encoder, and logs squared errors, which differ from the paper's stated setup in several places. Read ## Paper vs code before reimplementing.

Problem

Learned VIO that concatenates visual and inertial features assumes both are reliable. Missing or corrupted frames, noisy or biased IMU readings, and camera-IMU misalignment or time skew all degrade it, and the authors report that direct concatenation can underperform a vision-only model under IMU degradation § 1 § SI. They also note that classical filters handle corruption abruptly, failing completely in some of their tests § 4.6 tab. 5.

Method

The model has four parts: a visual encoder, an inertial encoder, a fusion function g and a temporal model with a pose regressor § 2 fig. 2. The visual encoder takes two consecutive frames stacked as six channels and uses a FlowNetS-style convolutional stack. Its last convolutional output is flattened and projected to a 256-dimensional vector § 2.1 eq. 1 § SI models.py train.py. The inertial encoder is a two-layer bidirectional LSTM with 128 hidden units per direction. It reads the IMU window attached to each frame pair and outputs a 256-dimensional vector § 2.1 eq. 2 § SI.

The fusion function combines the two vectors into one feature z § 2.2 eq. 3. The three options are:

  • Direct fusion: plain concatenation of the two vectors § 3.1 eq. 5.
  • Soft fusion: sigmoid masks computed from the concatenated features, one per modality, applied by element-wise multiplication § 3.2 eq. 6 eq. 7 eq. 8.
  • Hard fusion: a per-feature Bernoulli keep/drop decision, sampled with the Gumbel-max trick and relaxed with a temperature-controlled softmax so that gradients flow § 3.3 eq. 9 eq. 10 eq. 11 eq. 12 eq. 13 eq. 14 eq. 15 eq. 16 eq. 17.

The temporal model is a recurrent network that receives z for each frame pair and produces a state, and a linear head maps that state to a six-value pose change: three for translation, three for rotation § 2.3 eq. 4 § SI. Training uses a loss on the relative pose between consecutive frames train.py; the paper does not state a loss.

Equation numbers were counted from the LaTeX source, where each row of an eqnarray block takes a number; the compiled PDF was not available to check them. If a number seems off, cite the section instead.

Pseudocode
# Hard selective fusion for one frame pair  [§ 3.3] [eq. 14]
# Input:  visual (B, 256), inertial (B, 256), temperature tau
# Output: fused feature (B, 512)

def hard_fuse(visual, inertial, tau):
    # Concatenate the two modalities  [§ 2.2]
    joint = concat(visual, inertial)           # (B, 512)

    # Two logits per feature: drop or keep  [§ 3.3] [code: models.py#L248-L253]
    logits = keep_drop_mlp(joint)              # (B, 512, 2)

    # Sample a one-hot choice with Gumbel noise; gradients use the soft
    # version (straight-through)  [§ 3.3] [code: models.py#L217-L235]
    choice = gumbel_softmax_st(logits, tau)    # (B, 512, 2)

    # Keep the "keep" channel as the binary mask  [code: models.py#L270]
    mask = choice[:, :, 1]                     # (B, 512)
    return joint * mask
Pseudocode
# Training step on one sequence of frames  [§ 4.1] [code: train.py#L290-L394]
# Input:  frames, imu windows, ground-truth poses
# Output: scalar loss for one optimiser update

def training_step(frames, imus, poses):
    # Hidden state of the temporal model is reset per sequence
    # [code: train.py#L298]
    temporal_state = zeros()

    total_t = 0
    total_r = 0
    for j in range(n_pairs):
        # Visual features from the frozen encoder  [code: train.py#L322-L326]
        visual = fc_visual(encoder(frames[j], frames[j+1]))
        # Inertial features from the window attached to frame j
        # [code: train.py#L309] [§ 2.1]
        inertial = bilstm_imu(imus[j])
        # Fusion; tau is the current Gumbel temperature  [code: train.py#L350]
        fused = hard_fuse(visual, inertial, tau)
        # Temporal update and pose head  [§ 2.3]
        temporal_state = rnn(fused, temporal_state)
        pred = pose_head(temporal_state)           # (6,)
        # Relative ground truth between frames j and j+1
        # [code: train.py#L727-L734]
        true_t, true_r = relative_pose(poses[j], poses[j+1])
        total_t += mse(true_t, pred[:3])
        total_r += mse(true_r, pred[3:])

    # Weighted sum: rotation error is multiplied by 100
    # [code: train.py#L383]
    return total_t / n_pairs + 100 * total_r / n_pairs

Hyperparameters

NameValueWhereNotes
OptimiserAdam§ 4.1 train.py
Learning rate1e-4§ 4.1 train.pysame for all networks; no schedule in code
Adam betas0.9, 0.999train.pypaper gives them only as a commented-out line in the source; code is the only stated source
Weight decay0train.pynot stated in paper
Batch size8§ 4.1code default is 4 train.py; last short batch is skipped train.py
Epochsnot statedpapercode default 100 train.py; checkpoint saved every epoch train.py
Sequence lengthnot statedpapercode default 3 frames train.py
Input image size512 × 256§ SIcode has no resize step; size depends on the preprocessed data (see Paper vs code) data_loader.py
Visual encoder initFlowNetS pretrained on FlyingChairs§ SIcheckpoint path pretrain/flownets_EPE1.951.pth, not in repo train.py
Visual encoder trainingfrozen in shipped codetrain.pypaper says end-to-end; see Paper vs code
Visual feature size32×1024 → 256§ SI train.py
IMU window length10 samples per frame§ SI data_loader.pypaper: IMU at 100 Hz, images at 10 Hz § 4.2
IMU encoderBiLSTM, 2 layers, 128 hidden per direction, dropout 0.2§ 2.1 § SI models.pyoutput 256 = last step, both directions; SI lists an FC 128 layer that the code does not have
Temporal modelLSTM, 2 layers, 512 hidden, dropout 0.2§ 2.3 § SI models.py train.pypaper says bidirectional; code is unidirectional
Pose headtwo linear layers, 3 outputs each§ SI models.py
Soft masklinear 512→512 + sigmoid§ 3.2 models.py
Hard masklinear 512→512, ReLU, linear 512→1024, ReLU, 2-way Gumbel§ SI models.pySI lists one FC 1024 with sigmoid; code differs
Gumbel temperature, trainingstarts 1.0 each epoch, ×exp(−0.0006·i) every 10 batches, floor 0.5train.py train.pypaper: τ not given a value § 3.3
Gumbel temperature, evaluation0.5train.py
Loss weight on rotation100train.pynot stated in paper
Degradation probabilities, vision10% occlusion, 10% blur, 10% missing§ 4.4 data_loader.pyper frame
Degradation probabilities, all5% for each of seven types§ 4.4 data_loader.pyper frame
Occlusion patch128×128 at a random position§ 4.3.1 data_loader.py
Blur15 pixels§ 4.3.1paper says Gaussian σ=15; code uses a 15×15 mean filter data_loader.py
Salt-and-pepper amountnot statedpapercode uses 0.004 of pixels data_loader.py
IMU noise / biasas described in paper§ 4.3.2code values: accel +U(0.1, 0.2), gyro +U(0.001, 0.002) per sample data_loader.py
Spatial misalignmentup to 10 degrees§ 4.3.3code: integer 5–9 used as radians, z-axis only data_loader.py
Temporal misalignmentnot statedpapercode: each IMU sample zeroed with probability 0.5 data_loader.py
Train / test split, KITTItrain 00,01,02,04,06,08,09; test 05,07,10§ 4.2sequence 03 excluded
Train / test split, EuRoCtrain all except MH_04_difficult; test MH_04_difficult§ 4.2 § SIimages and IMU downsampled to 10 Hz and 100 Hz § 4.2
Train / test split, PennCOSYVIOtrain as, bf, bs; test af§ 4.2 § SIcamera 10 Hz, IMU 100 Hz
Hardwareone NVIDIA Titan X GPU§ 4.1
FrameworkPyTorch§ 4.1

Results

Author-reported. Setup: KITTI odometry (train on 7 sequences, test on 05, 07, 10), EuRoC MH_04_difficult, PennCOSYVIO af; monocular images with IMU; baselines are a vision-only network (DeepVO) and direct concatenation (VINet) § 4.1 § 4.2 § 4.4. Classical baselines are MSCKF on KITTI and OKVIS on EuRoC § 4.6. Each cell in the tables below is translation error and rotation error as the paper gives them.

Corrupted data, KITTI, one degradation at a time (Table 1 caption: translation in metres, rotation in degrees; the paper's caption says "absolute" error). The paper marks the best value per column and per metric in bold; that marking is not reproduced here tab. 1:

ModelOcclusionBlurMissing (vision)Noise and bias (IMU)Missing (IMU)SpatialTemporal
Vision Only0.117, 0.1480.117, 0.1530.213, 0.4560.116, 0.1360.116, 0.1360.116, 0.1360.116, 0.136
VIO Direct0.116, 0.1100.117, 0.1070.191, 0.1550.118, 0.1150.118, 0.1630.119, 0.1370.120, 0.111
VIO Soft0.116, 0.1050.119, 0.1040.198, 0.1490.119, 0.1050.118, 0.1290.119, 0.1280.119, 0.108
VIO Hard0.112, 0.1260.114, 0.1100.187, 0.1590.114, 0.1200.115, 0.1400.111, 0.1460.113, 0.133

KITTI, normal data and combined degradation (values as in tab. 2; the caption gives no units):

ModelNormalVision degradationAll degradation
Vision Only0.116, 0.1360.177, 0.3550.142, 0.281
VIO Direct0.116, 0.1060.175, 0.1640.148, 0.139
VIO Soft0.118, 0.0980.173, 0.1500.152, 0.134
VIO Hard0.112, 0.1100.172, 0.1510.145, 0.150

EuRoC MH_04_difficult, same columns tab. 3:

ModelNormalVision degradationAll degradation
Vision Only0.00976, 0.08670.0222, 0.2680.0190, 0.213
VIO Direct0.00765, 0.05400.0181, 0.06960.0162, 0.0935
VIO Soft0.00848, 0.05640.0170, 0.05330.0152, 0.0860
VIO Hard0.00795, 0.05890.0177, 0.05650.0157, 0.0823

PennCOSYVIO af, same columns tab. 4:

ModelNormalVision degradationAll degradation
Vision Only0.0379, 1.7550.0446, 1.8490.0414, 1.875
VIO Direct0.0377, 1.3500.0396, 1.2230.0407, 1.353
VIO Soft0.0381, 1.2520.0399, 1.1660.0405, 1.296
VIO Hard0.0387, 1.2960.0410, 1.2060.0400, 1.232

Classical methods, for comparison; "Fail" is the paper's word tab. 5:

DatasetNormalFull visual degradationOcclusion + blurFull sensor degradation
KITTI (MSCKF)0.116, 0.044Fail2.4755, 0.0726Fail
EuRoC (OKVIS)0.0283, 0.04020.0540, 0.05910.0198, 0.0400Fail

Claims in the text: soft fusion improves rotation on KITTI and EuRoC, and hard fusion improves translation § 4.4 § 4.5. In hand-held data, the gap in translation between fusion variants is small, and hard fusion is the best under combined degradation § 4.5. Learned fusion degrades gracefully while classical filters fail abruptly § 4.6. In the interpretation study, visual features are kept more than 60% of the time overall; the two modalities are kept equally often under occlusion; inertial features dominate when images are missing (over 90%) § 4.7 fig. 5. Selection also shifts with motion: inertial features are kept more during fast rotation and visual features more during fast translation § 4.7 fig. 6.

Paper vs code

Official code: changhao-chen/selective_sensor_fusion at commit 113eb8df88c59fe6c8dd2f0b472d6772018074b8. It is linked from neither the paper nor its project page in a way we could verify: the project page changhaoc.github.io/selective_sensor_fusion returned 404 when fetched. The repository's README names the paper and all seven authors and calls itself the webpage for the CVPR19 paper. Its commits are by the login changhao-chen and by the git name changhaoc. Ownership is therefore inferred, not confirmed. The repository has no LICENSE file, so no verbatim code is quoted here; all code is cited by path and line only.

The repository contains model code, a training script and a dataset loader. It has no evaluation script, no configuration files, and no pretrained weights. The README links a preprocessed dataset on a third-party file host, which we did not use.

Differences between paper and code:

  • Visual encoder: the paper says the network is end-to-end trainable, with the visual encoder initialised from FlyingChairs weights § 2 § SI. The shipped script runs FlowNet in evaluation mode without gradients, so only the 256-unit projection on top of it is trained train.py.
  • Convolution count: the paper's text says nine convolutional layers § 2.1. The code defines ten, adding a stride-1 layer conv6_1 with 1024 channels models.py. The supplementary table lists nine § SI.
  • Bidirectionality of the temporal model: the paper says bidirectional § 2.3 and the supplementary table says bidirectional, 512 hidden § SI. The code uses a unidirectional LSTM models.py train.py.
  • Inertial encoder: the supplementary table lists an FC 128 layer before the LSTM § SI. The code feeds the six raw IMU channels directly to the LSTM models.py train.py.
  • Hard mask: the paper defines the Bernoulli parameters with a sigmoid § 3.3 and the supplementary table lists a 1024-unit FC layer with sigmoid § SI. The code uses an extra 512-unit layer, ReLU activations, and a two-way softmax over logits with Gumbel noise models.py models.py.
  • Hard mask at test time: the mask sampler has no evaluation branch, so validation and test results also draw random masks models.py train.py train.py.
  • Which fusion is trained: the shipped script hard-codes hard fusion train.py. Soft and direct fusion require editing that constant. Degradation is also hard-coded off (mode 0) for training, validation and test train.py. The degraded-data results therefore need a code change to reproduce. The paper does not say whether training used corrupted data.
  • Batch size: paper 8 § 4.1; code default 4 train.py.
  • Gaussian blur: paper σ = 15 § 4.3.1; code uses a box filter of size 15 data_loader.py.
  • Missing images: paper removes 10% of frames § 4.3.1; code replaces them with zero images, keeping the frame count data_loader.py.
  • IMU noise: paper adds white noise to the accelerometer and a fixed bias to the gyroscope § 4.3.2. The code adds a fresh uniform offset to each sample for both sensors, with positive mean data_loader.py.
  • Spatial misalignment: paper says up to 10 degrees § 4.3.3. The code draws an integer between 5 and 9 and passes it to cos and sin as radians, rotating about the z axis only data_loader.py.
  • Temporal misalignment: paper describes a time shift between image and IMU windows § 4.3.3. The code zeroes each IMU sample with probability 0.5 data_loader.py.
  • IMU window alignment: the paper describes the IMU window as lying between two images § 2.1. The code attaches the window of the reference frame to each pair train.py.
  • Image size: the supplementary material says images are resized to 512×256 § SI. The code has no resize step; the only 512×256 array is the zero replacement for missing frames data_loader.py.
  • Metrics: the paper reports average absolute error, in metres and degrees tab. 1 § 4.4. The code logs mean squared errors for translation and for Euler angles in radians train.py train.py. The script that produces the paper's numbers is not in the repository.
  • Weight initialisation: the paper does not state one. The code applies Xavier initialisation to the pose head only; the other layers keep PyTorch defaults, though several modules define an unused Xavier initialiser train.py models.py.
  • FlowNet batch-norm flag: the script passes the batch size as the batchNorm argument, so batch normalisation is enabled and the convolutions have no bias train.py models.py. The paper does not mention batch normalisation.

Pitfalls

  • The gate is random at test time in the shipped hard model. Results vary with the random seed and with the number of samples drawn; fix the seed and report variance if you reproduce the table.
  • The Gumbel temperature is reset to 1.0 at the start of each epoch and then decays with the batch index, not a global step. From my reading of the code, it reaches the floor of 0.5 after roughly 150 batches each epoch train.py train.py.
  • The KITTI, EuRoC and PennCOSYVIO loaders depend on preprocessing that is not in the repository. Frame and IMU alignment, IMU sample counts per frame, and image sizes are assumed by the code. The paper says the IMU window has ten samples § SI; the code hard-codes ten only for the IMU-missing degradation data_loader.py.
  • Degradation patterns are drawn once when the dataset object is built, so every epoch sees the same corrupted samples data_loader.py.
  • The code uses NumPy and SciPy calls that newer releases removed or deprecated (np.int, scipy.misc.imread), and torch.rand(...).cuda() in the Gumbel sampler, so a CPU-only run fails data_loader.py data_loader.py models.py. The FlowNet batch norm flag described above is a silent surprise.
  • Salt-and-pepper noise is applied with list-of-arrays indexing into a three-dimensional image data_loader.py. Its effect depends on the NumPy version; I did not run it.
  • The paper states no loss, no epoch count, no sequence length and no checkpoint-selection rule. The code saves every epoch and the paper does not say which checkpoint it reports train.py.
  • The test loop assigns sequence labels 5, 7 and 10 by position in test.txt, not by reading the sequence name train.py.
  • Costs: the visual encoder runs on every frame pair under no-gradient mode, so training is cheaper than full end-to-end training. Per-pair sampling makes each forward pass depend on a random draw.

Lineage

  • Builds on: FlowNetS for the visual encoder § 2.1; the inertial LSTM design of IONet § 2.1; the Gumbel-max and Gumbel-softmax relaxations § 3.3; attention-style re-weighting § 3.2.
  • Compared against: DeepVO (vision-only), VINet (direct concatenation), MSCKF and OKVIS (classical) § 4.1 § 4.6.
  • Superseded by: not stated.

Source and license

Adapted from “Selective Sensor Fusion for Neural Visual-Inertial Odometry” by Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, Niki Trigoni (arXiv:1903.01534), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/changhao-chen/selective_sensor_fusion at commit 113eb8df88c5 (none stated (no LICENSE file in the repository)). Any quoted code is shown under that license.