Optical flow

Overview

Optical flow gives each pixel of one frame a 2D displacement to where it appears in the next frame. The cards use it in seven roles. Three estimate it by matching: GMFlow matches every pixel against every other one 2111.13680 §3.1, FlowFormer summarises the cost volume with a transformer before decoding 2203.16194 §3.2, and Unimatch uses one matching step for flow, stereo and depth 2211.05783 §3.1. One computes flow inside a SLAM tracker, where it feeds a bundle adjustment and is not output 2210.13641 §3.1. Two use an estimate inside a video model to align neighbouring frames before they are combined 2104.13371 §3.2 1903.10128 §3.1. One replaces flow with frame differences 2012.10071 §3.2. Four use flow only in training. Two build training data 1702.00871 §4 2203.03570 §3, one predicts flow images as an extra output of a robot policy 2512.18007 §3.2, and one uses flow from a separate estimator as a pseudo-label for a 3D motion head 2509.24209 §3.4.

The three matching estimators share one definition. The flow of a pixel is its matched position in the other frame minus its own position [arxiv:2111.13680 §3.1 eq. 3-4]. Unimatch uses the same definition 2211.05783 §3.1 eq. 4. FlowFormer's decoder works with the same matched position, p = x + f(x) 2203.16194 §3.3. The variants differ in how they find the match. Earlier regression networks search a local window and repeat small updates to reach distant matches, which makes large motion slow to resolve 2111.13680 §1. GMFlow searches the whole frame in one pass, and Unimatch keeps that pass for flow. FlowFormer keeps a local 9×9 window in its decoder, but each source pixel's tokens come from its full cost map 2203.16194 §3.2-3.3.

Variants

Global matching (GMFlow)

Each frame is turned into features at 1/8 resolution. Six Transformer blocks let the two feature maps attend to each other, in 2×2 local windows 2111.13680 §3.2. Every frame-1 pixel is then compared with every frame-2 pixel in one correlation, and a softmax over frame 2 gives a distribution over matches 2111.13680 §3.1 eq. 1-2. A self-attention step carries confident flow into pixels that have no match, such as occluded ones 2111.13680 §3.3. An optional refinement stage repeats the matching locally, in a 9×9 window, at 1/4 resolution 2111.13680 §3.4. Card: GMFlow.

Author-reported: with one refinement, the Things validation EPE is 2.80, against 4.25 for RAFT with 31 refinements 2111.13680 tab. 3. On Sintel test (clean) the EPE is 1.74, against 1.94 for RAFT [arxiv:2111.13680 tab. 5]. The method is weaker on KITTI test, 9.32 F1-all against 5.10 for RAFT 2111.13680 tab. 6. The paper attributes most of that gap to occluded pixels 2111.13680 §4.4.

Cost: the full correlation matrix has one row and one column per pixel 2111.13680 §3.1. The paper reports that computing it in blocks is cheap in time: 59.5 ms with 8×8 splits against 52.6 ms unsplit at 448×1024 input 2111.13680 app. B.

Shared matching for flow, stereo and depth (Unimatch)

Unimatch runs one matching step for three outputs, with one set of learned weights. Flow uses the full correlation over pixel pairs, as GMFlow does. Stereo uses the same softmax along one scanline. Depth runs a plane sweep over inverse-depth candidates, using the known cameras [arxiv:2211.05783 §3.1 eq. 1-13]. The matching layers have no learned parameters, so one network serves all three tasks 2211.05783 §3.1. A self-attention layer spreads matched values to unmatched pixels 2211.05783 §3.3. An optional refinement at 1/4 resolution repeats the match locally, in a 9×9 window for flow 2211.05783 §3.4.1. Card: Unifying Flow, Stereo and Depth Estimation.

Author-reported: the final flow model scores 1.03 EPE on Sintel test clean and 2.37 on final 2211.05783 tab. 14. On KITTI 2015 test it scores 4.49 on all pixels, after fine-tuning on KITTI 2012 and 2015 2211.05783 tab. 15 §4.5.1. These rows use the final model, not the base model of the GMFlow card. For stereo, the KITTI 2015 test D1-all is 1.77, and LEAStereo is listed at 1.65 2211.05783 tab. 17. On ScanNet depth, DeepV2D has a slightly lower Abs Rel than GMDepth, 0.057 against 0.059 2211.05783 tab. 21.

Cost: flow still builds a score for every pair of pixels. At the released 384×512 training crop, the 1/8 grid has 3,072 positions, so one pair needs about 9.4 million scores. This is arithmetic from the crop size, not a paper figure; the paper gives no memory figure 2211.05783 §3.1. Starting stereo from the flow model gives 2.58 Things EPE with no fine-tuning, against 75.24 from random initialisation 2211.05783 tab. 10. The paper reports no gain when stereo or depth models are transferred to flow §4.4.

Compressed cost volume with a transformer decoder (FlowFormer)

Each source pixel's cost map, a full H×W array of dot products, is reduced by three strided convolutions and summarised into K = 8 tokens of width D = 128 2203.16194 §3.2 tab. 2. Three alternate-group transformer layers then mix the tokens within a pixel and across pixels [arxiv:2203.16194 §3.2 eq. 2-3]. A recurrent decoder reads a 9×9 window of the cost map around each pixel's current match, cross-attends to that pixel's tokens, and updates the flow 2203.16194 §3.3 eq. 4-5. Card: FlowFormer.

Author-reported: Sintel test AEPE is 1.16 clean and 2.09 final, against 1.39 and 2.47 for GMA with warm start 2203.16194 tab. 1. On KITTI-15 test it reports 4.68 F1-all, second to S-Flow's 4.64, and this number uses tiling 2203.16194 tab. 1 app. §2. In the ablation, which uses the RAFT-style schedule, three AGT layers lower Sintel train clean from 1.29 to 1.10 over the same encoder without them 2203.16194 tab. 2.

Cost: the model has 18.2M parameters, against 5.9M for GMA [arxiv:2203.16194 tab. 3]. The transformer works on the summary, but the released code builds the full cost volume for each pair before summarising it, so the summary does not reduce that memory 2203.16194 §3.1. The main text reports no runtime or memory figures 2203.16194 §4.

Flow inside a SLAM tracker (NeRF-SLAM)

The tracker in NeRF-SLAM is DROID-SLAM. It computes dense optical flow between frame pairs with a RAFT-like network, and a convolutional GRU outputs a flow and a per-measurement weight at each step. A dense bundle adjustment then solves for keyframe poses and inverse-depth maps 2210.13641 §3.1. Flow here is a measurement inside an optimisation. The system does not output it. The mapper receives each keyframe's pose, image, depth and covariances, and no flow 2210.13641 §3.3. A radiance field is then trained with a depth term weighted by the marginal covariance of each depth value 2210.13641 §3.2 eq. 4. Card: NeRF-SLAM.

Author-reported on Replica: the average PSNR is 41.40 dB, against 17.76 dB for Nice-SLAM without depth. The average depth L1 is 4.49 cm, against 4.08 cm for Nice-SLAM with ground-truth depth, so its average depth error is higher 2210.13641 tab. 1. In the synthetic ablation, unweighted dense depth makes PSNR about 3 dB worse than using no depth, and weighting by covariance removes that loss 2210.13641 §4.4. The paper reports about 10 frames per second at 640 × 480 on one RTX 2080 Ti 2210.13641 §4.5.

Cost: the paper reports about 11 GB of GPU memory and names dense correlation volumes as one cause 2210.13641 §5. Its keyframe gate is a mean flow of 2.5 px, but the released code uses 2.4 px and a second distance check, so tune both thresholds together 2210.13641 §3.3. The card's Paper vs code section lists the details.

Flow-guided deformable alignment (BasicVSR++)

Here flow does not leave the model. Flow from frame i to the two earlier frames warps their feature states 2104.13371 §3.2 eq. 3. The offsets of a deformable convolution are the flow plus a learned residue, so flow is the base and the network only learns a correction to it [arxiv:2104.13371 §3.2 eq. 6]. The flow comes from a pre-trained SPyNet estimator, and the paper trains it with a lower learning rate than the rest of the model 2104.13371 §4. Card: BasicVSR++.

Author-reported: in the ablation, adding flow-guided alignment raises REDS4 PSNR from 31.48 to 31.94 dB, before second-order propagation is added 2104.13371 tab. 3. Without flow the model reaches 27.44 dB and its training collapses; a loss that keeps offsets close to the flow reaches 30.22 dB 2104.13371 tab. 4. The full model scores 32.39 dB on REDS4 with 7.3M parameters and 77 ms at 180×320 input 2104.13371 tab. 1. The ablation tables do not name their dataset, and only the full-model value matches the REDS4 value 2104.13371 tab. 3.

Flow as one input per neighbour (RBPN)

RBPN stacks the target frame, one neighbour and a dense flow between them into an eight-channel input and extracts features from it 1903.10128 §3.1. A projection step mixes these neighbour features with the running high-resolution estimate and passes a revised estimate on to the next neighbour 1903.10128 §3.2 eq. 3-6. Flow is therefore an input to learned features, not a warp applied directly to the frames. Card: RBPN.

Author-reported: on SPMCS-11 at 4x, RBPN/6-PF scores 30.10 dB against 29.42 dB for VSR-DUF 1903.10128 tab. 8. In the SPMCS-32 ablation, removing the flow from RBPN/5 lowers PSNR from 31.54 to 31.36 dB 1903.10128 tab. 6. Runtime grows with context length: 0.141 s for RBPN/6-PF at 120×160 input, on hardware the paper does not name 1903.10128 app. A.4. The paper says the flows are precomputed 1903.10128 §3.1; the card's Paper vs code section records that the official code computes them for each sample.

Frame differences instead of flow (TDN)

TDN gives its motion branch RGB differences between neighbouring frames instead of a flow field 2012.10071 §3.2. The short-term module takes four differences from a small window of frames and adds the result to the frame feature 2012.10071 §3.2 eq. 1. The paper's motivation is that two-stream networks need optical flow, which is expensive to compute 2012.10071 §1. Card: TDN.

The output is a class label, not a per-pixel field, so this variant does not replace flow for tasks that need dense motion. Author-reported: 52.3% top-1 on Something-Something V1 with eight frames of ResNet-50, at 36 GFLOPs 2012.10071 tab. 2. The released code blends the motion feature with fixed weights of 0.5 and 0.5 for eight segments, not by plain addition 2012.10071 §3.2 eq. 1; see the card.

Flow as training data (1702.00871, Kubric)

Two cards use flow to build data rather than to run a model.

In Video Salient Object Detection via Fully Convolutional Networks, training videos are made from annotated still images. Each image is moved by a flow field, I'(x) = I(x + v(x)), where the field is built on superpixels and then smoothed 1702.00871 §4 eq. 5-8. The trained model reads two frames and a static map, and it computes no flow 1702.00871 §3.3. Author-reported ablation, MAE on FBMS: 9.27% with image-derived video alone, 7.65% for the full model, and 24.5% with real video alone [arxiv:1702.00871 tab. 2].

In Kubric, the renderer writes optical flow from its render passes, alongside depth, segmentation and object trajectories 2203.03570 §3. RAFT pretrained on Kubric data reaches 1.89 and 3.02 on Sintel clean and final, and 4.82 AEPE on KITTI-15. The same table gives 2.27, 3.76 and 7.63 for FlyingChairs and 2.08, 2.75 and 4.66 for AutoFlow 2203.03570 §4.2. So Kubric beats FlyingChairs on all three, and AutoFlow beats Kubric on Sintel final and KITTI-15. The Kubric rendering parameters were chosen by hand, AutoFlow's were learned for Sintel, and the training schedule is not stated 2203.03570 §4.2.

Flow as an auxiliary prediction target (motion head)

Here flow is neither an input nor a rendered label. A pi-series vision-language-action model gets a second head, a diffusion transformer that predicts the latent of a flow image for the next action window. The target is RAFT flow between the current frame and the frame one chunk later, coloured with an HSV mapping and encoded by a frozen VAE 2512.18007 §3.2-3.3 app. A.1 app. A.2. The two heads share the backbone and their losses are summed 2512.18007 §3.2 eq. 7. Only the action head runs at inference, so the action path is unchanged 2512.18007 §1 §4.3. Card: Robotic VLA Benefits from Joint Learning with Motion Image Diffusion.

Author-reported: with a pi0.5 backbone the LIBERO average is 97.5%, against 96.9% without the motion head 2512.18007 tab. 1. On RoboTwin easy mode, the pi0 backbone averages 58.0% against 45.1% 2512.18007 tab. 2. In the LIBERO ablation, a motion-image target beats a future-image target on both backbones: 94.7% against 93.6% for pi0, and 97.5% against 95.7% for pi0.5 2512.18007 tab. 3. Each simulation policy is run with 50 or 100 rollouts, and no seed or interval is reported 2512.18007 §4.2 §4.3.

Flow as a pseudo-label for a 3D motion head (Forge4D)

Forge4D predicts dense 3D motion for each Gaussian, but its training data has no motion labels. It trains the motion head with two losses instead. A retargeting loss moves frame t's Gaussians to frame t−1 with the predicted motion, renders them, and compares the result with frame t−1 [arxiv:2509.24209 §3.4 eq. 6]. A flow loss matches the predicted 2D scene flow to flow from SEA-RAFT, a separate optical-flow model. A cyclic-consistency weight near zero in occluded regions removes those pixels §3.4 eq. 7. Flow is therefore a training signal only, and it reaches the model as a pseudo-label. Card: Forge4D.

Author-reported ablation: removing the flow loss lowers novel-time and novel-view PSNR from 29.04 to 28.87 dB, and removing the retargeting loss lowers it to 28.41 dB 2509.24209 tab. 7. Dense motion is scored only on the synthetic MetaHuman4D set, so the paper reports no real-data motion accuracy app. B tab. 5. The paper writes the flow loss with an L2 norm; the released code uses an L1 mean and a weight of 10 that the paper does not give 2509.24209 §3.4 eq. 7. The released stage-3 training loop exits before its optimisation loop runs, so the stage does not train as shipped (see the card's Paper vs code section).

When to use which

Implementation sketch

The global matching core, simplified to one refinement-free pass. Its reference card is GMFlow.

# Global matching at 1/8 resolution  [arxiv:2111.13680 §3.1 eq. 1-4]
# Input:  feat_1, feat_2 (D, h, w), with h = H/8 and w = W/8
# Output: flow (2, h, w), in 1/8-resolution pixels

def global_flow(feat_1, feat_2):
    # Score every frame-1 pixel against every frame-2 pixel  [eq. 1]
    q = flatten(feat_1)                       # (hw, D)
    k = flatten(feat_2)                       # (hw, D)
    corr = q @ transpose(k) / sqrt(D)         # (hw, hw), the memory cost
    # Distribution over frame-2 positions for each frame-1 pixel  [eq. 2]
    prob = softmax(corr, dim=-1)
    # Expected match position minus own position is the flow  [eq. 3-4]
    grid = pixel_grid(h, w)                   # (hw, 2)
    flow = prob @ grid - grid
    return reshape(flow, (2, h, w))

The alignment step of flow-guided deformable alignment, with one earlier frame instead of the paper's two. The card for BasicVSR++ has the full second-order form.

# Flow-guided alignment of an earlier state  [arxiv:2104.13371 §3.2]
# Inputs: feat (C, H, W), prev (C, H, W), flow (2, H, W) from i to i-1
# Output: aligned feature (C, H, W)

def align(feat, prev, flow):
    # Warp the earlier state onto frame i with the flow  [eq. 3]
    warped = warp(prev, flow)
    # Predict a residue to the flow, and a modulation mask  [eq. 6]
    residue, mask = offset_mask_net(feat, warped)
    # The flow is the base of the offsets; the network adds a correction
    offset = flow + residue
    # Sample the unwarped earlier state with these offsets  [eq. 5, 7]
    return modulated_deform_conv(prev, offset, sigmoid(mask))

Points where the papers and their official code differ are listed on the cards. For GMFlow, the flow-propagation formula and the loss normalisation differ from the paper GMFlow card. For BasicVSR++, the code bounds the residue and swaps the flow channels, and the paper's two-step flow is undefined there BasicVSR++ card. FlowFormer's decoder, RBPN's flow computation and TDN's fusion weights also differ from their papers; see each card's Paper vs code section before porting one. Two newer cards also matter for a port. Unimatch's released propagation layer adds learned query and key projections that the paper's equation 15 omits Unimatch card. Forge4D's stage-3 loop exits before its optimisation loop runs, and it releases no checkpoints Forge4D card.

Contradictions

• KITTI test numbers. GMFlow reports 9.32 F1-all against 5.10 for RAFT on KITTI test 2111.13680 tab. 6. FlowFormer reports 4.68 on the same test set, with tiling, against 5.10 for RAFT 2203.16194 tab. 1. The GMFlow card does not state the training mixture behind its KITTI test row, so the cards cannot tell whether the gap comes from the method or from the protocol.
• KITTI test, GMFlow family. GMFlow reports 9.32 F1-all on KITTI test 2111.13680 tab. 6. Unimatch's final flow model reports 4.49 on all pixels for KITTI 2015 test, after KITTI fine-tuning 2211.05783 tab. 15 §4.5.1. These are different models and benchmark rows, and the GMFlow card does not state its KITTI training mixture, so they are not a like-for-like conflict.
• FlowFormer on Sintel test final. FlowFormer reports 2.09 [arxiv:2203.16194 tab. 1]. Unimatch's table lists FlowFormer at 2.12 on the same split 2211.05783 tab. 14. The cards do not explain the difference, so compare the two only with that in view.
• RAFT baseline depends on warm start. On Sintel test clean, RAFT scores 1.94 in GMFlow's table 2111.13680 tab. 5 and in FlowFormer's table without warm start, but 1.61 with warm start 2203.16194 tab. 1. Compare across the two cards only with that protocol in view.
• FlowFormer's own numbers. The same model is reported at 1.01 and 2.40 on Sintel train clean and final in Table 1, and at 1.00 and 2.45 in Table 3 2203.16194 tab. 1 tab. 3. The card records the same stated schedule for both.
• Value of flow inside a video model. BasicVSR++ reports that training without flow collapses, at 27.44 dB 2104.13371 tab. 4. RBPN reports 31.36 dB without flow against 31.54 dB with it 1903.10128 tab. 6. The architectures, datasets and metrics differ, and the BasicVSR++ ablation does not name its dataset, so these are not a like-for-like conflict.

The estimator numbers are author-reported and are not directly comparable across the cards, because each paper uses its own protocol.

Papers

• Video Salient Object Detection via Fully Convolutional Networks: flow-warped synthetic video as training data for a model that computes no flow.
• Recurrent Back-Projection Network for Video Super-Resolution: per-neighbour flow as an input to recurrent back-projection.
• NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields: DROID-SLAM's flow inside a bundle adjustment, with covariance-weighted depth for a radiance field.
• TDN: Temporal Difference Networks for Efficient Action Recognition: RGB frame differences as a flow-free motion cue for action recognition.
• BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment: flow as the base of deformable alignment in a recurrent video model.
• GMFlow: Learning Optical Flow via Global Matching: one-pass global matching with Transformer-enhanced features and flow propagation to unmatched pixels.
• Unifying Flow, Stereo and Depth Estimation: one parameter-free matching step, with shared weights, for flow, stereo and depth.
• Kubric: A scalable dataset generator: a renderer that writes exact optical flow for pretraining and other tasks.
• FlowFormer: A Transformer Architecture for Optical Flow: a transformer that reads a compressed cost volume and decodes flow recurrently.
• Robotic VLA Benefits from Joint Learning with Motion Image Diffusion: RAFT flow images as an auxiliary training target for a vision-language-action model.
• Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos: flow from a separate estimator as a pseudo-label for a 3D motion head.