Paper · 2019-03-04 · cs.CV, cs.AI, cs.LG, cs.RO
Selective Sensor Fusion for Neural Visual-Inertial Odometry
Monocular visual-inertial odometry with learned masks that re-weight concatenated visual and IMU features. Soft (sigmoid) and hard (Gumbel-softmax) masks beat direct concatenation on corrupted KITTI, EuRoC and PennCOSYVIO data, with smaller gains on clean data. Results are author-reported.
Visual-inertial odometryMultimodal fusionsensor-fusionrobustnessodometry
arXivPDFCodeMarkdown13 min read
AI-compiled by claude-haiku-5-5 on 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A camera and an inertial sensor (accelerometer and gyroscope) can both tell a moving device how it has moved. The camera is good at direction and turning but cannot measure distance travelled at true scale. The inertial sensor is fast and works in the dark, but it drifts over time. Real sensors also fail: frames get dropped, lenses get dusty, and clocks disagree. Many learned systems simply stack the two streams together and hope for the best, so a bad camera feed can drag the estimate off course.
This paper adds a gate between the two streams. A small network looks at both sets of features and decides, for each feature, how much to let through. The gate is trained together with the rest of the network. The authors try two kinds of gate: a smooth one that scales each feature between zero and one, and a stricter one that either passes a feature or blocks it. Both are compared with plain concatenation on three public datasets, including corrupted versions of them. The gates help most when the input is damaged. Looking at the gate values also shows which sensor the network trusts in a given situation, for example the inertial stream during fast turns.
TL;DR
- Monocular visual-inertial odometry (VIO) with a learned fusion step that re-weights visual and IMU features before a recurrent pose regressor § 2 § 3.
- Soft fusion uses sigmoid masks; hard fusion samples binary masks with Gumbel-softmax so it trains end to end § 3.2 § 3.3.
- On corrupted data the gated variants beat direct concatenation on most columns, while clean-data gains are small or absent tab. 2 tab. 3 tab. 4.
- The released code trains only the hard variant, freezes the visual encoder, and logs squared errors, which differ from the paper's stated setup in several places. Read
## Paper vs codebefore reimplementing.
Problem
Learned VIO that concatenates visual and inertial features assumes both are reliable. Missing or corrupted frames, noisy or biased IMU readings, and camera-IMU misalignment or time skew all degrade it, and the authors report that direct concatenation can underperform a vision-only model under IMU degradation § 1 § SI. They also note that classical filters handle corruption abruptly, failing completely in some of their tests § 4.6 tab. 5.
Method
The model has four parts: a visual encoder, an inertial encoder, a fusion
function g and a temporal model with a pose regressor § 2 fig. 2. The
visual encoder takes two consecutive frames stacked as six channels and uses
a FlowNetS-style convolutional stack. Its last convolutional output is
flattened and projected to a 256-dimensional vector § 2.1 eq. 1
§ SI models.py train.py. The inertial encoder
is a two-layer bidirectional LSTM with 128 hidden units per direction. It reads
the IMU window attached to each frame pair and outputs a 256-dimensional
vector § 2.1 eq. 2 § SI.
The fusion function combines the two vectors into one feature z
§ 2.2 eq. 3. The three options are:
- Direct fusion: plain concatenation of the two vectors § 3.1 eq. 5.
- Soft fusion: sigmoid masks computed from the concatenated features, one per modality, applied by element-wise multiplication § 3.2 eq. 6 eq. 7 eq. 8.
- Hard fusion: a per-feature Bernoulli keep/drop decision, sampled with the Gumbel-max trick and relaxed with a temperature-controlled softmax so that gradients flow § 3.3 eq. 9 eq. 10 eq. 11 eq. 12 eq. 13 eq. 14 eq. 15 eq. 16 eq. 17.
The temporal model is a recurrent network that receives z for each frame
pair and produces a state, and a linear head maps that state to a six-value
pose change: three for translation, three for rotation § 2.3 eq. 4
§ SI. Training uses a loss on the relative pose between consecutive frames
train.py; the paper does not state a loss.
Equation numbers were counted from the LaTeX source, where each row of an
eqnarray block takes a number; the compiled PDF was not available to check
them. If a number seems off, cite the section instead.
# Hard selective fusion for one frame pair [§ 3.3] [eq. 14]
# Input: visual (B, 256), inertial (B, 256), temperature tau
# Output: fused feature (B, 512)
def hard_fuse(visual, inertial, tau):
# Concatenate the two modalities [§ 2.2]
joint = concat(visual, inertial) # (B, 512)
# Two logits per feature: drop or keep [§ 3.3] [code: models.py#L248-L253]
logits = keep_drop_mlp(joint) # (B, 512, 2)
# Sample a one-hot choice with Gumbel noise; gradients use the soft
# version (straight-through) [§ 3.3] [code: models.py#L217-L235]
choice = gumbel_softmax_st(logits, tau) # (B, 512, 2)
# Keep the "keep" channel as the binary mask [code: models.py#L270]
mask = choice[:, :, 1] # (B, 512)
return joint * mask# Training step on one sequence of frames [§ 4.1] [code: train.py#L290-L394]
# Input: frames, imu windows, ground-truth poses
# Output: scalar loss for one optimiser update
def training_step(frames, imus, poses):
# Hidden state of the temporal model is reset per sequence
# [code: train.py#L298]
temporal_state = zeros()
total_t = 0
total_r = 0
for j in range(n_pairs):
# Visual features from the frozen encoder [code: train.py#L322-L326]
visual = fc_visual(encoder(frames[j], frames[j+1]))
# Inertial features from the window attached to frame j
# [code: train.py#L309] [§ 2.1]
inertial = bilstm_imu(imus[j])
# Fusion; tau is the current Gumbel temperature [code: train.py#L350]
fused = hard_fuse(visual, inertial, tau)
# Temporal update and pose head [§ 2.3]
temporal_state = rnn(fused, temporal_state)
pred = pose_head(temporal_state) # (6,)
# Relative ground truth between frames j and j+1
# [code: train.py#L727-L734]
true_t, true_r = relative_pose(poses[j], poses[j+1])
total_t += mse(true_t, pred[:3])
total_r += mse(true_r, pred[3:])
# Weighted sum: rotation error is multiplied by 100
# [code: train.py#L383]
return total_t / n_pairs + 100 * total_r / n_pairsHyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Optimiser | Adam | § 4.1 train.py | |
| Learning rate | 1e-4 | § 4.1 train.py | same for all networks; no schedule in code |
| Adam betas | 0.9, 0.999 | train.py | paper gives them only as a commented-out line in the source; code is the only stated source |
| Weight decay | 0 | train.py | not stated in paper |
| Batch size | 8 | § 4.1 | code default is 4 train.py; last short batch is skipped train.py |
| Epochs | not stated | paper | code default 100 train.py; checkpoint saved every epoch train.py |
| Sequence length | not stated | paper | code default 3 frames train.py |
| Input image size | 512 × 256 | § SI | code has no resize step; size depends on the preprocessed data (see Paper vs code) data_loader.py |
| Visual encoder init | FlowNetS pretrained on FlyingChairs | § SI | checkpoint path pretrain/flownets_EPE1.951.pth, not in repo train.py |
| Visual encoder training | frozen in shipped code | train.py | paper says end-to-end; see Paper vs code |
| Visual feature size | 32×1024 → 256 | § SI train.py | |
| IMU window length | 10 samples per frame | § SI data_loader.py | paper: IMU at 100 Hz, images at 10 Hz § 4.2 |
| IMU encoder | BiLSTM, 2 layers, 128 hidden per direction, dropout 0.2 | § 2.1 § SI models.py | output 256 = last step, both directions; SI lists an FC 128 layer that the code does not have |
| Temporal model | LSTM, 2 layers, 512 hidden, dropout 0.2 | § 2.3 § SI models.py train.py | paper says bidirectional; code is unidirectional |
| Pose head | two linear layers, 3 outputs each | § SI models.py | |
| Soft mask | linear 512→512 + sigmoid | § 3.2 models.py | |
| Hard mask | linear 512→512, ReLU, linear 512→1024, ReLU, 2-way Gumbel | § SI models.py | SI lists one FC 1024 with sigmoid; code differs |
| Gumbel temperature, training | starts 1.0 each epoch, ×exp(−0.0006·i) every 10 batches, floor 0.5 | train.py train.py | paper: τ not given a value § 3.3 |
| Gumbel temperature, evaluation | 0.5 | train.py | |
| Loss weight on rotation | 100 | train.py | not stated in paper |
| Degradation probabilities, vision | 10% occlusion, 10% blur, 10% missing | § 4.4 data_loader.py | per frame |
| Degradation probabilities, all | 5% for each of seven types | § 4.4 data_loader.py | per frame |
| Occlusion patch | 128×128 at a random position | § 4.3.1 data_loader.py | |
| Blur | 15 pixels | § 4.3.1 | paper says Gaussian σ=15; code uses a 15×15 mean filter data_loader.py |
| Salt-and-pepper amount | not stated | paper | code uses 0.004 of pixels data_loader.py |
| IMU noise / bias | as described in paper | § 4.3.2 | code values: accel +U(0.1, 0.2), gyro +U(0.001, 0.002) per sample data_loader.py |
| Spatial misalignment | up to 10 degrees | § 4.3.3 | code: integer 5–9 used as radians, z-axis only data_loader.py |
| Temporal misalignment | not stated | paper | code: each IMU sample zeroed with probability 0.5 data_loader.py |
| Train / test split, KITTI | train 00,01,02,04,06,08,09; test 05,07,10 | § 4.2 | sequence 03 excluded |
| Train / test split, EuRoC | train all except MH_04_difficult; test MH_04_difficult | § 4.2 § SI | images and IMU downsampled to 10 Hz and 100 Hz § 4.2 |
| Train / test split, PennCOSYVIO | train as, bf, bs; test af | § 4.2 § SI | camera 10 Hz, IMU 100 Hz |
| Hardware | one NVIDIA Titan X GPU | § 4.1 | |
| Framework | PyTorch | § 4.1 |
Results
Author-reported. Setup: KITTI odometry (train on 7 sequences, test on 05, 07, 10), EuRoC MH_04_difficult, PennCOSYVIO af; monocular images with IMU; baselines are a vision-only network (DeepVO) and direct concatenation (VINet) § 4.1 § 4.2 § 4.4. Classical baselines are MSCKF on KITTI and OKVIS on EuRoC § 4.6. Each cell in the tables below is translation error and rotation error as the paper gives them.
Corrupted data, KITTI, one degradation at a time (Table 1 caption: translation in metres, rotation in degrees; the paper's caption says "absolute" error). The paper marks the best value per column and per metric in bold; that marking is not reproduced here tab. 1:
| Model | Occlusion | Blur | Missing (vision) | Noise and bias (IMU) | Missing (IMU) | Spatial | Temporal |
|---|---|---|---|---|---|---|---|
| Vision Only | 0.117, 0.148 | 0.117, 0.153 | 0.213, 0.456 | 0.116, 0.136 | 0.116, 0.136 | 0.116, 0.136 | 0.116, 0.136 |
| VIO Direct | 0.116, 0.110 | 0.117, 0.107 | 0.191, 0.155 | 0.118, 0.115 | 0.118, 0.163 | 0.119, 0.137 | 0.120, 0.111 |
| VIO Soft | 0.116, 0.105 | 0.119, 0.104 | 0.198, 0.149 | 0.119, 0.105 | 0.118, 0.129 | 0.119, 0.128 | 0.119, 0.108 |
| VIO Hard | 0.112, 0.126 | 0.114, 0.110 | 0.187, 0.159 | 0.114, 0.120 | 0.115, 0.140 | 0.111, 0.146 | 0.113, 0.133 |
KITTI, normal data and combined degradation (values as in tab. 2; the caption gives no units):
| Model | Normal | Vision degradation | All degradation |
|---|---|---|---|
| Vision Only | 0.116, 0.136 | 0.177, 0.355 | 0.142, 0.281 |
| VIO Direct | 0.116, 0.106 | 0.175, 0.164 | 0.148, 0.139 |
| VIO Soft | 0.118, 0.098 | 0.173, 0.150 | 0.152, 0.134 |
| VIO Hard | 0.112, 0.110 | 0.172, 0.151 | 0.145, 0.150 |
EuRoC MH_04_difficult, same columns tab. 3:
| Model | Normal | Vision degradation | All degradation |
|---|---|---|---|
| Vision Only | 0.00976, 0.0867 | 0.0222, 0.268 | 0.0190, 0.213 |
| VIO Direct | 0.00765, 0.0540 | 0.0181, 0.0696 | 0.0162, 0.0935 |
| VIO Soft | 0.00848, 0.0564 | 0.0170, 0.0533 | 0.0152, 0.0860 |
| VIO Hard | 0.00795, 0.0589 | 0.0177, 0.0565 | 0.0157, 0.0823 |
PennCOSYVIO af, same columns tab. 4:
| Model | Normal | Vision degradation | All degradation |
|---|---|---|---|
| Vision Only | 0.0379, 1.755 | 0.0446, 1.849 | 0.0414, 1.875 |
| VIO Direct | 0.0377, 1.350 | 0.0396, 1.223 | 0.0407, 1.353 |
| VIO Soft | 0.0381, 1.252 | 0.0399, 1.166 | 0.0405, 1.296 |
| VIO Hard | 0.0387, 1.296 | 0.0410, 1.206 | 0.0400, 1.232 |
Classical methods, for comparison; "Fail" is the paper's word tab. 5:
| Dataset | Normal | Full visual degradation | Occlusion + blur | Full sensor degradation |
|---|---|---|---|---|
| KITTI (MSCKF) | 0.116, 0.044 | Fail | 2.4755, 0.0726 | Fail |
| EuRoC (OKVIS) | 0.0283, 0.0402 | 0.0540, 0.0591 | 0.0198, 0.0400 | Fail |
Claims in the text: soft fusion improves rotation on KITTI and EuRoC, and hard fusion improves translation § 4.4 § 4.5. In hand-held data, the gap in translation between fusion variants is small, and hard fusion is the best under combined degradation § 4.5. Learned fusion degrades gracefully while classical filters fail abruptly § 4.6. In the interpretation study, visual features are kept more than 60% of the time overall; the two modalities are kept equally often under occlusion; inertial features dominate when images are missing (over 90%) § 4.7 fig. 5. Selection also shifts with motion: inertial features are kept more during fast rotation and visual features more during fast translation § 4.7 fig. 6.
Paper vs code
Official code: changhao-chen/selective_sensor_fusion at commit
113eb8df88c59fe6c8dd2f0b472d6772018074b8. It is linked from neither the
paper nor its project page in a way we could verify: the project page
changhaoc.github.io/selective_sensor_fusion returned 404 when fetched. The
repository's README names the paper and all seven authors and calls itself the
webpage for the CVPR19 paper. Its commits are by the login changhao-chen
and by the git name changhaoc. Ownership is therefore inferred, not
confirmed. The repository has no LICENSE file, so no verbatim code is quoted
here; all code is cited by path and line only.
The repository contains model code, a training script and a dataset loader. It has no evaluation script, no configuration files, and no pretrained weights. The README links a preprocessed dataset on a third-party file host, which we did not use.
Differences between paper and code:
- Visual encoder: the paper says the network is end-to-end trainable, with the visual encoder initialised from FlyingChairs weights § 2 § SI. The shipped script runs FlowNet in evaluation mode without gradients, so only the 256-unit projection on top of it is trained train.py.
- Convolution count: the paper's text says nine convolutional layers § 2.1.
The code defines ten, adding a stride-1 layer
conv6_1with 1024 channels models.py. The supplementary table lists nine § SI. - Bidirectionality of the temporal model: the paper says bidirectional § 2.3 and the supplementary table says bidirectional, 512 hidden § SI. The code uses a unidirectional LSTM models.py train.py.
- Inertial encoder: the supplementary table lists an FC 128 layer before the LSTM § SI. The code feeds the six raw IMU channels directly to the LSTM models.py train.py.
- Hard mask: the paper defines the Bernoulli parameters with a sigmoid § 3.3 and the supplementary table lists a 1024-unit FC layer with sigmoid § SI. The code uses an extra 512-unit layer, ReLU activations, and a two-way softmax over logits with Gumbel noise models.py models.py.
- Hard mask at test time: the mask sampler has no evaluation branch, so validation and test results also draw random masks models.py train.py train.py.
- Which fusion is trained: the shipped script hard-codes hard fusion train.py. Soft and direct fusion require editing that constant. Degradation is also hard-coded off (mode 0) for training, validation and test train.py. The degraded-data results therefore need a code change to reproduce. The paper does not say whether training used corrupted data.
- Batch size: paper 8 § 4.1; code default 4 train.py.
- Gaussian blur: paper σ = 15 § 4.3.1; code uses a box filter of size 15 data_loader.py.
- Missing images: paper removes 10% of frames § 4.3.1; code replaces them with zero images, keeping the frame count data_loader.py.
- IMU noise: paper adds white noise to the accelerometer and a fixed bias to the gyroscope § 4.3.2. The code adds a fresh uniform offset to each sample for both sensors, with positive mean data_loader.py.
- Spatial misalignment: paper says up to 10 degrees § 4.3.3. The code draws an integer between 5 and 9 and passes it to cos and sin as radians, rotating about the z axis only data_loader.py.
- Temporal misalignment: paper describes a time shift between image and IMU windows § 4.3.3. The code zeroes each IMU sample with probability 0.5 data_loader.py.
- IMU window alignment: the paper describes the IMU window as lying between two images § 2.1. The code attaches the window of the reference frame to each pair train.py.
- Image size: the supplementary material says images are resized to 512×256 § SI. The code has no resize step; the only 512×256 array is the zero replacement for missing frames data_loader.py.
- Metrics: the paper reports average absolute error, in metres and degrees tab. 1 § 4.4. The code logs mean squared errors for translation and for Euler angles in radians train.py train.py. The script that produces the paper's numbers is not in the repository.
- Weight initialisation: the paper does not state one. The code applies Xavier initialisation to the pose head only; the other layers keep PyTorch defaults, though several modules define an unused Xavier initialiser train.py models.py.
- FlowNet batch-norm flag: the script passes the batch size as the
batchNormargument, so batch normalisation is enabled and the convolutions have no bias train.py models.py. The paper does not mention batch normalisation.
Pitfalls
- The gate is random at test time in the shipped hard model. Results vary with the random seed and with the number of samples drawn; fix the seed and report variance if you reproduce the table.
- The Gumbel temperature is reset to 1.0 at the start of each epoch and then decays with the batch index, not a global step. From my reading of the code, it reaches the floor of 0.5 after roughly 150 batches each epoch train.py train.py.
- The KITTI, EuRoC and PennCOSYVIO loaders depend on preprocessing that is not in the repository. Frame and IMU alignment, IMU sample counts per frame, and image sizes are assumed by the code. The paper says the IMU window has ten samples § SI; the code hard-codes ten only for the IMU-missing degradation data_loader.py.
- Degradation patterns are drawn once when the dataset object is built, so every epoch sees the same corrupted samples data_loader.py.
- The code uses NumPy and SciPy calls that newer releases removed or
deprecated (
np.int,scipy.misc.imread), andtorch.rand(...).cuda()in the Gumbel sampler, so a CPU-only run fails data_loader.py data_loader.py models.py. TheFlowNetbatch norm flag described above is a silent surprise. - Salt-and-pepper noise is applied with list-of-arrays indexing into a three-dimensional image data_loader.py. Its effect depends on the NumPy version; I did not run it.
- The paper states no loss, no epoch count, no sequence length and no checkpoint-selection rule. The code saves every epoch and the paper does not say which checkpoint it reports train.py.
- The test loop assigns sequence labels 5, 7 and 10 by position in
test.txt, not by reading the sequence name train.py. - Costs: the visual encoder runs on every frame pair under no-gradient mode, so training is cheaper than full end-to-end training. Per-pair sampling makes each forward pass depend on a random draw.
Lineage
- Builds on: FlowNetS for the visual encoder § 2.1; the inertial LSTM design of IONet § 2.1; the Gumbel-max and Gumbel-softmax relaxations § 3.3; attention-style re-weighting § 3.2.
- Compared against: DeepVO (vision-only), VINet (direct concatenation), MSCKF and OKVIS (classical) § 4.1 § 4.6.
- Superseded by: not stated.
Source and license
Adapted from “Selective Sensor Fusion for Neural Visual-Inertial Odometry” by Changhao Chen, Stefano Rosa, Yishu Miao, Chris Xiaoxuan Lu, Wei Wu, Andrew Markham, Niki Trigoni (arXiv:1903.01534), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/changhao-chen/selective_sensor_fusion at commit 113eb8df88c5 (none stated (no LICENSE file in the repository)). Any quoted code is shown under that license.