Un-EVIMO: Unsupervised Event-based Independent Motion Segmentation

In plain words

Most systems that learn to spot moving objects need people to draw masks around them. This paper skips that step. An event camera does not take frames: each pixel reports when its brightness changes, which makes fast motion easy to follow.

The trick is geometry. If you know how far each point in the scene is, and how the camera itself moved, you can predict how the background should move in the image. Anything that moves differently from that prediction must be a separate object. The method fits the camera's motion to the bulk of the scene while ignoring the outliers, subtracts that motion, and turns the large leftover motion into rough masks.

Those rough masks train a small network that sees only events. At test time it needs no depth and no camera data. The authors report results close to some supervised methods on the EVIMO dataset. Their masks have less sharp edges than the supervised ones.

TL;DR

• Segments independently moving objects from events alone, with no motion masks in training.
• Labels come from a geometric teacher: depth, flow, RANSAC camera-motion fit, Otsu threshold on the residual.
• Student: 15-bin event volume into a U-Net with an ImageNet ResNet encoder, trained with focal loss.
• Author-reported event-masked IoU on EVIMO: 50 (table), 45 (box), 56 (floor), 53 (wall), 44 (fast).
• Depth is used only in training. The code is MIT, but several training settings differ from the paper: read "Paper vs code" first.

Problem

Event-based independent-motion segmentation has relied on labelled event masks §2. Unsupervised alternatives fit a mixture of several parametric motion models, which requires choosing the number of models and tuning per scene §3. The paper argues that the lack of labelled event data is the bottleneck and that a per-pixel classifier can be trained if its labels come from geometry §3§4.

Method

The pipeline has a label-generating teacher and a student that sees only events §4fig. 1.

1. Predict dense event flow with an E-RAFT network, initialised from DSEC weights and fine-tuned to match RAFT flow computed on the grayscale frames §4.1app. §3.5.
2. For each pixel, the rigid flow induced by the camera is linear in six camera-motion values, given depth and pixel coordinates eq. 7.
3. Fit those six values with RANSAC, so that independently moving pixels are outliers §4.2.
4. Compute the camera-induced background flow and the residual flow between predicted and background flow §4.3.
5. Split the residual magnitudes with Otsu's method on a clipped histogram §4.3eq. 8eq. 9.
6. Keep a training example only if its residual statistics look confident §4.3.
7. Train the student, a U-Net on a 15-bin event volume, to reproduce the labels §4.5.

Motion model underlying the teacher. The observed flow is split into a camera term and per-object terms, with each point belonging to at most one object eq. 4eq. 5. The residual is the part not explained by the camera.

Per-pixel block of the camera-motion system (the paper stacks one x-row and one y-row per pixel; the column order is the six camera-motion values) eq. 7:

\begin{bmatrix} -1/z & 0 & x/z & xy & -(1+x^2) & y \\ 0 & -1/z & y/z & 1+y^2 & -xy & -x \end{bmatrix} eq. 7

Otsu objective over candidate thresholds r_j, with P_k the bin probabilities of the clipped residual histogram eq. 8eq. 9:

\arg\max_{r_j} \sum_{k=0}^{r_j} P_k\,(\mu_{bg}(r_j)-\mu)^2 + \sum_{k=r_j}^{K_{max}} P_k\,(\mu_{imo}(r_j)-\mu)^2 eq. 8

Teacher, one frame. Lines marked "released code only" are not in the paper.

# Pseudo-label for one frame  [§4] [fig. 1]
# Input:  flow (H, W, 2) in px, depth (H, W) in m, intrinsics K
# Output: binary label (H, W), accepted flag

def pseudo_label(flow, depth, K):
    # Valid pixels  [code: unevimo/teacher.py#L287-L290]
    valid = is_finite(flow) and is_finite(depth) and depth > 0
    # Build the two rows per pixel from depth and coordinates  [eq. 7]
    system = rigid_motion_rows(depth, K)           # (H, W, 2, 6)
    # Fit six camera-motion values, treating IMO pixels as outliers  [§4.2]
    motion, inliers = ransac(system[valid], flow[valid])
    # Rigid flow each pixel would have from camera motion alone  [§4.2]
    background = system @ motion                   # (H, W, 2)
    # Residual flow and its magnitude  [§4.3]
    residual = flow - background
    magnitude = norm(residual, axis=-1)
    # Otsu split of residuals clipped at 10 px, 256 bins  [§4.3] [eq. 8]
    threshold = otsu(magnitude[valid], bins=256, upper=10.0)
    label = valid and magnitude > threshold
    # Confidence checks on the residual histogram  [§4.3]
    accepted = residual_checks(magnitude, label, valid)
    return label, accepted

# Released code only  [code: tools/generate_labels.py#L92-L100]
def released_postprocess(label, inliers, depth):
    # Clear pixels that RANSAC explained as camera motion
    label = label and not inliers
    # Zero the label beyond 2.5 m, after the accept decision
    label = label and depth <= 2.5
    return label

Student, one event slice. The student sees events only; depth and camera data are used only to build labels §4.5app. §3.6.

# Student: events in, IMO probability out  [§4.5] [app. §3.6]
# Input:  events of one slice
# Output: probability map (256, 256)

def student(events):
    # Splat events into 15 bins  [§4.5] [code: unevimo/voxel.py#L35-L69]
    volume = event_volume(events, bins=15)         # (15, 215, 320)
    # Resize to 256 x 256 with nearest neighbour  [app. §3.6]
    volume = resize_nearest(volume, 256, 256)
    # U-Net, ImageNet ResNet encoder, one output  [§4.5] [app. §3.6]
    logits = unet(volume)                          # (1, 256, 256)
    # Sigmoid gives the IMO probability per pixel  [app. §3.6]
    return sigmoid(logits)

Focal loss used in the released training command. Alpha and gamma are not stated in the paper §4.5.

# Focal loss, alpha 0.25, gamma 2  [code: student_seg/train_student.py#L68-L84]
def focal_loss(prob, target):
    # Per-pixel binary cross-entropy on probabilities
    bce = binary_cross_entropy(prob, target)
    # Probability assigned to the true class
    pt = exp(-bce)
    # Down-weight pixels the network already predicts well
    return 0.25 * (1 - pt) ** 2 * bce

Hyperparameters

Results

Author-reported. Setup: EVIMO sequences, event-masked IoU at 40 Hz §5eq. 13. Baselines are supervised networks trained with ground-truth masks, plus EMSGC, an unsupervised graph-cut method §5app. §3.2.

Event-masked IoU (mean ± std, percent) §5:

Optical-flow end-point error against RAFT flow on held-out sequences, pixels §4.1:

Camera pose from flow and depth, RANSAC §4.2:

Runtime per event slice, milliseconds §5. Hardware: one RTX 3080 mobile GPU and an 8-core CPU.

Ablation on the flow model used for labels, IoU per scene §6:

Detection rate (IoU at least 0.3) per scene, from the supplement app. §2: Wall 0.853, Table 0.817, Floor 0.912, Box 0.703, Fast 0.694.

Source notes: the IoU table is discussed in §5; the flow table is in §4.1; the pose table is in §4.2; the timing table is in §5; the ablation is in §6. These numbers are copied from the paper. None were re-run.

Paper vs code

Official code: https://github.com/ZiyunClaudeWang/un-evimo at commit 81fcbafbae2fb0f30b84b29cd603711128ebe2dc, MIT licence. The paper's project page links it, and the owner is the first author's account. Checked: teacher, flow training, student training, evaluation, pipeline scripts and README.

Teacher and labels:

• Released pipeline clears RANSAC-inlier pixels from the label generate_labels.py. The paper does not describe this step.
• Released pipeline zeroes labels beyond 2.5 m after the accept decision generate_labels.pygenerate_labels.py. The paper says depth up to 3 m is used app. §3.1.
• Released pipeline disables the Otsu separability test (0.0) generate_labels.py. The teacher's own default is 0.5 teacher.py.
• The paper's first confidence test discards a slice when the error variance is above a threshold §4.3. The code rejects when the normalised residual variance is below 0.2 teacher.py. The statistic is also computed on clipped, mean-normalised residuals teacher.py. Direction and statistic both differ.
• The paper's second confidence test uses the variance between object and background pixels §4.3. The code uses a normalised Otsu between-class ratio teacher.py.
• RANSAC trial limit 300 and stop probability 0.999 match the paper §4.2teacher.py. The inlier threshold of 1.0 px and the seed are not in the paper teacher.py.
• Camera motion uses inverse depth with a small epsilon floor teacher.py. The paper states no floor.
• Otsu uses 256 bins over 0 to 10 px, matching the paper §4.3teacher.py.

Flow network:

• The paper says the flow network is fine-tuned for 10 epochs app. §3.5. The released command runs 6 epochs at learning rate 2e-4 and batch 16 2_train_flow.sh. The script's default is 10 train_event_flow.py.
• The paper gives no flow optimiser. The code uses AdamW, weight decay 1e-5 and a one-cycle schedule train_event_flow.pytrain_event_flow.py.
• The paper says flow is supervised by RAFT flow with "photometric matching and refinement" §4.1. The training loss is an exponentially weighted L1 over 12 refinement iterations against RAFT flow, with no photometric term train_event_flow.py.
• Validation for the flow network holds out floor seq_01 and fast seq_00 train_event_flow.py. The paper does not state a split.

Student:

• Training sequences conflict within the paper. Section 4.5 lists table, wall, floor, box and fast §4.5. The supplement lists table, wall, floor and box app. §3.1. The code trains on box, floor, wall, table and fast, excludes tabletop in the released command 4_train_student.sh, and holds out one sequence per scene for validation train_student.py.
• Encoder conflicts within the paper: ResNet34 in §4.5 and ResNet18 in the supplement §4.5app. §3.6. The code's default is resnet18 train_student.py.
• The released command enables flips, polarity inversion and rotation up to 90 degrees 4_train_student.shtrain_student.py. The paper does not mention augmentation.
• The focal loss in the code uses alpha 0.25 on every term and sums over pixels, divided by batch size train_student.pytrain_student.py. The paper gives no parameters. Without the --focal-loss flag the code uses the segmentation library's focal loss instead train_student.py.
• Model selection uses the lowest validation loss, checked every 200 steps, and never the ground-truth masks train_student.pytrain_student.py. The paper says it trained until the validation loss curve flattened app. §3.6.
• Validation drops frames whose ground-truth object has fewer than 500 pixels train_student.pytrain_student.py. The supplement says all slices with at least one object are evaluated app. §3.1.
• The event volume is normalised per sample over non-zero cells voxel.py. The paper does not mention this.
• Polarity is mapped to −1 or +1 before splatting, so the two event polarities have opposite signs voxel.py.

Evaluation:

• Decision threshold 0.4 in the code eval_student.pyeval_student.py. The paper states none.
• The paper's published table averages IoU per frame; the repository README says so and reports pooled pixel IoU for its own table README.md. The eval script defaults to pooled pixels eval_student.py.
• The repository's own numbers differ from the paper: 48, 45, 61, 56, 47 with mean 51.5 README.md. Use the paper's numbers for claims.
• Temporal smoothing (exponential moving average over flow-warped probabilities, alpha 0.3) is code only and not in the paper eval_temporal.pyREADME.md. The README reports mean 55.1 with it.
• The README reports teacher-label quality (mean 46.5, precision 0.578, recall 0.723). The paper does not report this README.md.

Paper internal issues to know before implementing:

• Equation 7 has a typo in the 6-DOF block: "n + y_n^2" should read 1 + y_n^2. The code uses 1 + y^2 teacher.py.
• Equation 12 has a typo: "y^n_n" should be y_n.
• The 6-DOF parametric variant is named in the ablation but never defined; the paper defines only the 12-DOF biquadratic flow §4.4§6.
• The ablation text does not say whether variants (b) and (c) use input depth. The text in §4.4 says parametric flow can be used without depth.
• The pose table's caption calls the translation error a mean squared error, but its units are metres. The text gives the Fast error as "4" with no unit; the table gives 0.0416 m §4.2.
• The EMSGC comparison selects the best top-K percentile of IoU per sequence app. §3.2. That selection uses ground-truth IoU, so the "unsupervised" baseline is tuned on evaluation labels.
• The flow comparison uses RAFT flow as ground truth, not true flow §4.1.
• EMSGC is better than the paper's method on the table scene: 55 against 50 §5.
• Timing hardware is an RTX 3080 mobile GPU §5; training used an RTX 3090 app. §3.6.

Pitfalls

• Depth dependence at training time. The teacher needs depth, flow and camera intrinsics. The paper names this as a dependency it leaves for future work §6. Depth from the EVIMO dataset has holes, and the pipeline fills them with morphology app. §3.1.
• Object motion can bias the camera fit. Without outlier rejection, the camera-motion fit drifts toward near and fast-moving objects §4.2. Keep the RANSAC inlier step in any reimplementation.
• Thresholds are data-dependent. Otsu on the residual gives a per-frame threshold, which the paper motivates by variable noise and depth §4.3. Test it on your own data before reusing the confidence gates.
• Slice-level inference. Each event slice is predicted on its own. Objects that barely move, or that move during a slice, are missed or truncated §7app. §1.
• Temporal inconsistency. Predictions flicker across consecutive slices. The paper suggests temporal constraints as future work §7.
• Boundaries. The student's masks lack the sharp edges of supervised ones. The paper attributes this to noisy pseudo-labels and to labels computed at one instant rather than over a time window §5app. §1.
• Reproducibility of EMSGC numbers. The paper's EMSGC numbers come from the best of several K values and per-sequence tuning, not from a fixed configuration app. §3.2.
• Licence. Repository is MIT. The event-flow model files carry no licence header; cite them, do not copy them.

Lineage

• Builds on: E-RAFT for event-based flow §4.1; RAFT for image flow used as the distillation target §4.1; Otsu thresholding §4.3; RANSAC §4.2; U-Net with ImageNet-pretrained encoder and focal loss §4.5. None of these is compiled here.
• Compared against: EMSGC (spatio-temporal graph cuts, unsupervised) §5; EVIMO (supervised, EVIMO dataset) §5; SpikeMS (supervised spiking network) §5; GConv and EVDodgeNet (supervised) §5.
• Superseded by: not known from this wiki.