VideoMamba: State Space Model for Efficient Video Understanding

In plain words

Video is long and repetitive, so a video model has to look at many frames without spending huge amounts of memory. Older designs either look only at small neighbourhoods of pixels and frames (3D convolutions) or let every token attend to every other token (video transformers), and the second option gets expensive quickly as clips get longer.

VideoMamba replaces attention with a state-space model called Mamba. A state-space model passes a compact memory along the sequence, so its cost grows in a straight line with the length of the input. The video is cut into small spatiotemporal patches, read as one sequence, and processed by a plain stack of the same kind of blocks used in Vision Transformers. Each block reads the sequence in both directions. The default token order reads one frame at a time.

Bigger versions of the model tend to overfit. The authors fix this by having a smaller, already trained copy of the model guide the bigger one during training. They report accuracy that is competitive with attention-based models on short and long video benchmarks, and much lower memory use on long clips.

TL;DR

• Adapts the bidirectional Mamba block (a linear-time state-space layer) to video, in a ViT-style stack with no downsampling, in three sizes: Ti, S and M §3.2 tab. 1.
• Scales the model with self-distillation from a smaller trained model, which the authors report avoids overfitting §3.3 §4.1.
• Pretrains with masked alignment to a CLIP ViT-B teacher for self-supervised video or video-text work §3.4 §4.2.
• Author-reported: 84.0% ImageNet-1K top-1 at 576px tab. 2; 83.3% Kinetics-400 at 64x384px tab. 3; 68.4% Something-Something V2 at 16x288px tab. 4.
• Code is official and Apache-2.0 licensed. Several paper and code details differ (see Paper vs code).

Problem

Video has large spatiotemporal redundancy inside short clips and complex dependencies across long ones §1. 3D CNNs and video transformers each address one of these, but not both §1. Attention costs grow quadratically with input length §3.2. Divided space-time attention reduces that cost but adds parameters and, per the authors, underperforms joint attention, especially with masked pretraining §3.2.

Mamba-style state-space models give linear-complexity sequence modelling §1. The original Mamba is built for 1D sequences and lacks spatial awareness for images, which Vision Mamba (Vim) addresses with a bidirectional block §3.1 fig. 2. The paper extends that bidirectional block to 3D video §3.1.

Assumptions: supervised ImageNet-1K pretraining for the Ti/S/M backbones §4.1; K400 and SSv2 fine-tuning from ImageNet weights §4.2; self-supervised video pretraining distilled from CLIP ViT-B §4.2; and, for long videos, end-to-end training on sparsely sampled frames tab. XIV (see Lineage).

Method

Overview. The model follows the ViT layout §3.2 fig. 3. A 3D convolution with kernel and stride (1, 16, 16) turns the clip into non-overlapping spatiotemporal tokens, L = t × h × w with h = H/16 and w = W/16 §3.2 eq. 7. A learnable class token is prepended. Learnable spatial positions p_s and temporal positions p_t are added §3.2 eq. 7. The tokens pass through L stacked bidirectional Mamba blocks, and the final class token, after normalisation, goes to a linear classifier §3.2.

Spatiotemporal scan. The 2D bidirectional scan is extended to 3D in three orders §3.2 fig. 4: spatial-first (organise by location, then stack frame by frame), temporal-first (organise by frame, then stack along space), and a hybrid. The hybrid has a v1 form that runs half of each and a v2 form that runs both fully, at 2× compute §3.2. The authors find spatial-first the most effective and simplest §3.2 fig. 7.

Bidirectional SSM. Each block runs a selective-scan SSM forward over the sequence and a second one on the reversed sequence, then adds the two outputs §3.1 mamba_simple.py. The state dimension is 16 and the expansion ratio is 2, the Mamba defaults §3.3.

Discretisation. The continuous state equations are discretised with zero-order hold §3.1 eq. 3 eq. 4, giving the recurrence h_t = Ā h_{t-1} + B̄ x_t and output y_t = C h_t eq. 5 eq. 6. In Mamba, B, C and the step size Δ are functions of the input §3.1.

Self-distillation. For each scale, a smaller trained model acts as teacher for the larger student: the M model uses a pretrained S model as teacher §4.1. The student's final feature maps are aligned with the teacher's by an L2 loss, added to the classification loss §4.1. The authors say this overcomes the overfitting seen when the pure Mamba model is scaled up §3.3 fig. 6.

Masked alignment pretraining. Video pretraining follows UMT §3.4. The student is trained from scratch on video data, with unmasked tokens aligned to CLIP ViT-B features. Only the final outputs are aligned, unlike UMT's multi-layer alignment §3.4. Video uses 80% masking §4.2. For multimodal pretraining, the model is paired with a text encoder and a BERT cross-modal decoder §3.4, trained on WebVid-2M and CC3M with vision-text contrast, vision-text matching, masked language modelling and unmasked token alignment §4.4.

Pseudocode for the encoder (paths below are relative to the repository root at code_commit):

# VideoMamba forward pass  [§3.2] [eq. 7]
# [code: videomamba/video_sm/models/videomamba.py#L310-L363]
# Input:  video (B, 3, T, H, W)
# Output: class logits (B, num_classes)

def videomamba(video):
    # Non-overlapping 3D patches, kernel = stride = (1, 16, 16)  [§3.2]
    patches = conv3d_patch_embed(video)            # (B, C, T, H/16, W/16)
    # One token per patch, so L = T * (H/16) * (W/16)  [§3.2]
    tokens = flatten_to_sequence(patches)          # (B, L, C)
    # Prepend a learnable class token  [§3.2]
    tokens = concat(class_token, tokens)           # (B, L + 1, C)
    # Add the spatial positions p_s to every token  [§3.2] [eq. 7]
    tokens = tokens + spatial_pos_embedding        # (B, L + 1, C)
    # Add the temporal positions p_t per frame  [§3.2]
    # [code: videomamba/video_sm/models/videomamba.py#L319-L323]
    tokens = add_temporal_embedding(tokens)        # (B, L + 1, C)
    # Stacked bidirectional Mamba blocks with residuals  [§3.2] [§3.1]
    for block in blocks:
        tokens = bidirectional_mamba_block(tokens)
    # Normalise and keep only the class token for the head  [§3.2]
    class_feature = norm(tokens)[:, 0]             # (B, C)
    return linear_head(class_feature)              # (B, num_classes)

Pseudocode for the bidirectional block and the self-distillation step:

# Bidirectional Mamba block  [§3.1]
# [code: mamba/mamba_ssm/modules/mamba_simple.py#L183-L211]
# Input and output: tokens (B, L, C)

def bidirectional_mamba_block(tokens):
    # Forward selective scan over the token order  [§3.1]
    forward_out = selective_scan(tokens)           # (B, L, C_inner)
    # Reverse the sequence, scan it, then reverse the output back  [§3.1]
    backward_out = selective_scan(reverse(tokens)) # (B, L, C_inner)
    backward_out = reverse(backward_out)           # (B, L, C_inner)
    # Sum both directions, then project back to the model width
    # [code: mamba/mamba_ssm/modules/mamba_simple.py#L211]
    return out_projection(forward_out + backward_out)

# Self-distillation step for the larger student  [§4.1]
# [code: videomamba/image_sm/engine_distill.py#L40-L50]
# Teacher: pretrained smaller VideoMamba, no gradient  [§4.1]

def distill_step(images, labels):
    # Teacher features for the same (already augmented) images  [§4.1]
    teacher_features = teacher(images)             # (B, C), normalised
    # Student logits and features  [§4.1]
    logits, student_features = student(images)     # (B, K), (B, C)
    # Align the final feature maps, see Paper vs code  [§4.1]
    align_loss = mean(2 - 2 * dot(student_features, teacher_features))
    return cross_entropy(logits, labels) + align_loss

Hyperparameters

Units: learning rates are per the stated base batch, which the code scales (see Paper vs code). "Table X" and "XI" to "XIV" are Appendix B tables.

Model sizes tab. 1 videomamba.py

SSM block

Masked pretraining, single-modality (K400 and SSv2 columns share these values, except the flip rule) tab. X §4.2

Masked pretraining, multimodal tab. X §4.4 app. B

ImageNet-1K supervised tab. XI §4.1

Kinetics-400 supervised tab. XII §4.2

Something-Something V2 supervised tab. XIII §4.2

Long-video: Breakfast, COIN, LVU tab. XIV §4.3

Self-distillation §4.1 run224.sh

Results

Author-reported throughout. Setup: ImageNet-1K for supervised image classification §4.1; Kinetics-400 and Something-Something V2 for short-term video §4.2; Breakfast, COIN and LVU for long-term video §4.3; WebVid-2M and CC3M pretraining with five benchmarks for zero-shot retrieval §4.4. Efficiency was measured on one NVIDIA A100-80G, PyTorch 2.1, CUDA 11.8, batch size 128, 224 px frames fig. 1. The speed and memory values are in the figure and are not transcribed here.

Ablations (SSv2, author-reported).

• Scan type (fig. 7): spatial-first bidirectional 65.1; alternating spatial and temporal 65.1; hybrid v2 64.2; hybrid v1 63.9; temporal-first 62.4 fig. 7.
• Mask type (tab. 5): attention 68.5; clip-row 68.2; frame-row 67.8; random 67.4; tube 66.3 tab. 5.
• Mask ratio (tab. 5): 80% 68.5 (best of 50%, 65%, 80%, 90%) tab. 5.
• Alignment layer (tab. 5): last layer 68.5; last 2 layers 68.4; last 6 layers 68.2 tab. 5.
• Drop path (tab. 5): 0.4 gives 68.5; 0.1 gives 68.0 tab. 5.
• Frames and resolution (fig. 7): more frames consistently help K400; that does not hold on SSv2, and higher resolution does not uniformly help video §4.2 fig. 7.
• Self-distillation and early stopping (fig. 6): plotted only; the source text gives no numbers. The authors say early stopping did not help §4.1 fig. 6.

The ablation table uses a 200-epoch distillation schedule tab. 5, while the main results use 800 epochs tab. X.

Paper vs code

The code is at github.com/OpenGVLab/VideoMamba, which the paper links in its author block. The owner is the authors' lab. Code paths are relative to the repository root at code_commit. I read the image and video model files, the image distillation engine and scripts, the video fine-tuning and pretraining scripts, and the Mamba block. I did not run the code, and I did not check the vendored mamba/ and causal-conv1d/ directories beyond the Mamba block. Their licences are not checked.

1. Temporal positions and the class token. The paper's eq. 7 adds p_t to the whole sequence after prepending the class token §3.2 eq. 7. The code adds the temporal embedding to the patch tokens only, and the class token is handled separately videomamba.py. The paper's shape for p_t (t × C) is also not consistent with adding it to the (L + 1)-token sequence as written.
2. Distillation loss. The paper says L2 alignment §4.1. The code computes 2 − 2·cos on L2-normalised student and teacher features, averaged, with weight 1 and no α engine_distill.py videomamba_distill.py. For unit vectors this equals the squared L2 distance, so "L2" means the squared distance between normalised features, not the raw one.
3. Distillation hyperparameters. The published M script uses 20 warmup epochs run224.sh. The text says 5 §4.1, and the table says 30 for M tab. XI. The script's minimum learning rate is 5e-6 [videomamba/image_sm/exp_distill/videomamba_middle/run224.sh#L27]; the table says 1e-5 tab. XI. The script also clips gradients at 5.0 [videomamba/image_sm/exp_distill/videomamba_middle/run224.sh#L30], which the paper does not mention.
4. ImageNet defaults. The text gives default learning rate 1e-3 and batch 1024 §4.1. The table gives 5e-4 and 512 tab. XI. The code default is 5e-4 main.py, scaled by batch × GPUs / 512 main.py.
5. Mixup. The hyperparameter tables list label smoothing and CutMix, not mixup tab. XI tab. XII. The code default mixup is 0.8 [code: videomamba/image_sm/main.py#L132] and 0.8 in the video fine-tuning script run_class_finetuning.py. I did not check whether each run script overrides it.
6. Fine-tuning learning-rate rule. The text gives lr = base × batch / 256 §4.2. The code multiplies by num_sample as well [code: videomamba/video_sm/run_class_finetuning.py#L524-L526]. The K400 masked script and the SSv2 script set num_sample to 2 [code: videomamba/video_sm/exp/k400/videomamba_middle_mask/run_f64x224to384.sh#L34] run_f16x224.sh. Whether the paper's "batch size" already counts the two clips is not stated.
7. Masked pretraining rate. This one matches. The script gives base 1.5e-4, batch 64 per GPU and 32 GPUs run_mask_pretrain.sh. The scaled rate is 1.5e-4 × 2048 / 256 = 1.2e-3, as in tab. X. The script runs 801 epochs; the table says 800 run_mask_pretrain.sh.
8. Model sizes. The Ti, S and M depth and width match the code videomamba.py. The text gives 74M for the 84.0% result, while tab. 2 gives 75M.
9. Teacher. The M model's teacher is VideoMamba-S, as the paper says §4.1 run224.sh.
10. FLOPs. The code says FLOPs counting needs rms_norm=False and fused_add_norm=False videomamba.py. The paper does not say so.

Pitfalls

• Mixed input sizes in headline claims. Several gains compare a higher-resolution VideoMamba row with a lower-resolution baseline. The text's "+2.0%" over DeiT-B compares 448 px (83.8) with DeiT-B at 224 px (81.8) §4.1 tab. 2. The "+2.0%" over ViViT-L on K400 compares 64x384 (83.3) with 16x224 (81.3) §4.2 tab. 3. The "+2.6%" over TimeSformer on K400 uses the 96-frame TimeSformer-L row (80.7) §3.2 tab. 3. The "+5.9%" on SSv2 compares the 16x288 row (68.4) with TimeSformer-HR (62.5) §3.2 tab. 4. Compare matched input sizes when you reproduce.
• Compute of the bidirectional scan. Both directions run in every block. The v2 hybrid scan costs 2× compute §3.2. The paper does not give a separate figure for the bidirectional block's cost. The 6× speed and 40× memory figures are for 64 frames on the authors' setup §1 fig. 1.
• Backward scan reverses everything. The backward branch flips the whole token sequence, so the frame order and the spatial order inside each frame are reversed mamba_simple.py.
• Scaling and overfitting. A larger pure Mamba model overfits and can be worse than S when trained from scratch §4.1 §3.3. Self-distillation needs a trained smaller model first, which is an extra training run. Early stopping did not help §4.1.
• Input resolution and frames. More frames help K400 but not SSv2, and higher resolution does not uniformly help video §4.2 fig. 7. The SSv2 best row (16x288) uses a higher resolution than the 16x224 row tab. 4.
• Masking details not stated. The paper does not say whether masked tokens are dropped or replaced by a learned token. The row and attention masks are described in words §3.4 fig. 5 §4.2, and their exact construction is not given. Check the masked-pretraining code before you implement either.
• Teacher dependency. Self-supervised video pretraining distils from CLIP ViT-B (CLIP-400M) §4.2 tab. 3. The CLIP weights and their licence are a separate dependency. This card does not check them.
• Different pretraining data in comparisons. Long-video baselines use features from models pretrained on Kinetics-600 (Swin-B, ViS4mer) or on IN-21K with HTM data (Distant Supervision) §4.3 tab. 6. VideoMamba is trained end-to-end on K400 §4.3. The gains are not like-for-like.
• Zero-shot retrieval is mixed. At 17M pairs, UMT reaches a higher MSRVTT R@1 (35.5) than VideoMamba (34.7) tab. 8, so the "superior to UMT" claim §4.4 holds only for some settings.
• Unmasked tuning is needed for multimodal use. The authors add one unmasked epoch at lr 8e-5 because of positional sensitivity §4.4 app. B.
• Not run. I did not run the model, the training or the fast Mamba kernels. The fast path depends on the Mamba CUDA kernels, which this card does not cover.

Lineage

• Builds on: Mamba's selective scan, covered by Mamba: Linear-Time Sequence Modeling with Selective State Spaces §3.1.
• Builds on: Vision Mamba (Vim), a bidirectional Mamba block for images. It has no card here, and its arXiv id is not in the source bibliography §2 §3.1 fig. 2. The paper says VideoMamba omits Vim's middle class token and rotary position embedding, and reports gains over Vim on ImageNet-1K §3.2.
• Builds on: VMamba (visual state-space model), which adds depthwise convolution and downsampling. It has no card here and no arXiv id in the source bibliography §2 §3.2.
• Builds on: ViT isotropic layout §3.2 fig. 3, DeiT training recipe §4.1, VideoMAE training recipe §4.2, and UMT masked alignment §3.4 §4.4. None has a card here.
• Compared against: TimeSformer, ViViT, UniFormer, VideoMAE, UMT, ViS4mer, Turbo and Distant Supervision §4 tab. 3 tab. 4 tab. 6 tab. 7 tab. 8.
• Superseded by: not known from content in this wiki.