FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling

In plain words

Judging how good a video looks is expensive when the model has to read every pixel of a high-definition clip. The obvious shortcut, shrinking the frame, blurs exactly the fine detail that tells a sharp video from a blurry one. Cutting out one corner loses the rest of the picture.

FAST-VQA takes a different route. It divides each frame into a grid, takes a small patch from every grid cell at full resolution, and glues those patches into one smaller picture. The same patch positions are reused in every frame, so motion such as shaking still shows up. A transformer then scores this picture. Two changes stop it from being confused by the seams between patches, which are not real damage in the video. The authors report that this is far cheaper than earlier methods on high-resolution video and that it is also more accurate.

TL;DR

• Fragments are the input: raw-resolution mini-patches from a uniform grid, aligned across frames and spliced into one canvas.
• A Video Swin-T backbone is modified in two places: gated relative position biases separate same-patch pairs from cross-patch pairs, and a regression head scores each patch before pooling.
• The whole model is trained end to end with a PLCC-based loss, instead of regressing from frozen features.
• Author-reported: SRCC 0.779 and PLCC 0.814 on LSVQ 1080p, 99.5% fewer FLOPs than PVQ at 1080P, and state-of-the-art transfer to five small VQA datasets.
• The paper does not state the optimiser, learning rate, schedule, epoch count or augmentation, so the training recipe cannot be reproduced from this card alone.

Problem

Deep video quality assessment (VQA) models get expensive on high-resolution video. A 1080×1920 clip needs 42.5 times the FLOPs of a 224×224 input in the authors' measurement §1 tab. 4. Existing methods therefore freeze a pretrained recognition or quality backbone and train only a small regressor on its features §1 §2. The authors argue that frozen features are not tuned for quality and limit accuracy §1.

Naive down-sampling is ruled out. Bilinear resizing removes local texture that quality depends on, and random cropping breaks the match between a crop's quality and the whole video's quality §1 §3.1. The paper's goal is a sampling scheme that keeps local quality, covers the whole frame uniformly, keeps context, and keeps inter-frame motion, so that end-to-end training becomes practical §1.

Method

3.1 Grid Mini-patch Sampling (GMS)

GMS builds the model input, called a fragment, from one video of T frames §3.1 fig. 4.

1. Uniform grid. Each frame is cut into G_f × G_f cells of equal size, where G_f is the grid count §3.1 eq. 1.
2. Raw-resolution patch. From each cell, one mini-patch of size S_f × S_f is taken at original resolution with no resizing. The paper calls this random patch sampling §3.1 eq. 2.
3. Temporal alignment. The sampling operation is shared across all T frames, so the patch in cell (i, j) sits at the same place in every frame §3.1 eq. 3.
4. Splicing. Each patch is placed at its own cell position in a canvas of size (G_f·S_f) × (G_f·S_f), per frame, keeping the spatial layout §3.1 eq. 4.

The result is a fragment video with T frames of canvas size §3.1 tab. 2. Figure 2 shows fragments in spatial and temporal views, and states that fragments keep local detail and show shake across frames fig. 2. Figure 4 lays out the GMS pipeline (grid partition, patch sampling, splicing and temporal alignment), after which the fragments feed FANet fig. 4.

3.2 Fragment Attention Network (FANet)

FANet uses Video Swin Transformer Tiny (Swin-T) with four hierarchical self-attention stages as its backbone §3.2 fig. 5. The authors chose a patch-wise hierarchical transformer because its operations are naturally local to patches §2 §3.2. Two modifications address the discontinuities between mini-patches.

Gated Relative Position Biases (GRPB). Swin-T adds a learned relative position bias T to each attention score. In a fragment, two tokens can have the same relative offset but very different real distances if they lie in different mini-patches §3.2 fig. 3. GRPB learns two tables instead of one. T^real serves same-patch pairs and T^pseudo serves cross-patch pairs §3.2. A binary gate G marks same-patch pairs (1 if the pair lies in one mini-patch, else 0). The modified score matrix is M_A = QK^T + G ⊗ B_In + (1 − G) ⊗ B_Cr §3.2 eq. 5 eq. 6.

Intra-Patch Non-Linear Regression (IP-NLR). Pooling features before regression would mix quality across patches §3.2 fig. 3. IP-NLR passes each patch's features through non-linear regression layers R_NL first, then pools the resulting scores: s_pred = Pool(R_NL(f)) §3.2 eq. 7.

Training objective

The loss is a differentiable PLCC-based loss, l = (1 − PLCC(s_pred, s_gt)) / 2 §4.1. The paper states a batch size of 16 §4.1.

Pseudocode

The pseudocode below follows the paper's equations. The sampling-offset distribution and the pooling operator are not stated in the paper, so those lines are marked.

# Fragment construction: Grid Mini-patch Sampling  [§3.1] [eq. 1-4]
# Input:  video (T, H, W, 3), raw resolution
# Output: fragment video (T, G * S, G * S, 3)
# G = grid count G_f, S = patch size S_f  [tab. 2]

def build_fragment(video, grid_count, patch_size):
    # Frame count and frame size  [§3.1]
    num_frames, height, width, _ = video.shape
    # One offset per cell, shared by all frames  [§3.1] [eq. 3]
    # The offset distribution is not stated; "random" is all the paper says
    offsets = sample_offsets(grid_count, height, width, patch_size)
    fragment = zeros(num_frames, grid_count * patch_size, grid_count * patch_size, 3)
    for row in range(grid_count):
        for col in range(grid_count):
            # Take an S x S raw-resolution patch from this cell  [eq. 2]
            top, left = offsets[row, col]
            patch = video[:, top:top + patch_size, left:left + patch_size]
            # Splice the patch at its cell position  [eq. 4]
            fragment[:, row * patch_size:(row + 1) * patch_size,
                     col * patch_size:(col + 1) * patch_size] = patch
    return fragment

# Gated relative position bias for one attention window  [§3.2] [eq. 5-6]
# Table shape and index mapping are not stated

def gated_attention_scores(queries, keys, same_patch, relative_index):
    # Raw attention scores  [eq. 6]
    scores = queries @ keys.transpose()
    # Bias for same-patch pairs from the real-position table  [eq. 5]
    real_bias = real_position_table[relative_index]
    # Bias for cross-patch pairs from the pseudo-position table  [eq. 5]
    pseudo_bias = pseudo_position_table[relative_index]
    # Gate: 1 when the two tokens lie in the same mini-patch  [§3.2]
    gate = same_patch
    # Combine the two biases with the gate  [eq. 6]
    scores = scores + gate * real_bias + (1 - gate) * pseudo_bias
    return scores

# Intra-Patch Non-Linear Regression head  [§3.2] [eq. 7]
# Pooling operator and regressor width are not stated

def ip_nlr_score(patch_features):
    # Regress each patch's features on its own  [§3.2]
    patch_scores = nonlinear_regressor(patch_features)
    # Pool after regression, not before  [eq. 7]
    score = pool(patch_scores)
    return score

# Training loss  [§4.1]
# Taken over a batch of 16; the paper does not say which batch the PLCC uses

def plcc_loss(predicted, target):
    # Pearson correlation between predictions and labels  [§4.1]
    correlation = pearson(predicted, target)
    # Map correlation in [-1, 1] to a loss in [0, 1]  [§4.1]
    return (1 - correlation) / 2

Two sampling densities

FAST-VQA and FAST-VQA-M use the same patch size and differ in frame count and grid count. FAST-VQA-M is the lower-density, lower-cost version §4.1 tab. 2.

Hyperparameters

Values are from the main text and Table 2. Every item marked "not stated" is absent from the compiled paper.

Results

Author-reported. All numbers below are from the paper's tables. Training is on LSVQ only, and KoNViD-1k and LIVE-VQC are cross-dataset tests unless noted §4.2.

Benchmark (SRCC / PLCC) tab. 3.

Improvement of FAST-VQA over PVQ w/ patch, as the table states: +6%, +6%, +10%, +10%, +9%, +8%, +7%, +5% in column order tab. 3. These are relative gains. On LSVQ 1080p PLCC the absolute gain is 0.814 − 0.739 = 0.075 (derived). The text says the accuracy gain on 1080p is 8.10% over the full-resolution Swin-T baseline on PLCC §4.2 tab. 3.

Efficiency: FLOPs (G) and time (s, GPU / CPU) tab. 4. The paper reports the average of ten runs per sample. Multipliers are relative to FAST-VQA.

Transfer after fine-tuning (SRCC / PLCC) tab. 5. Each target dataset is split at random 10 times, and the mean is reported §4.4.

The table's own "Improvements led by VQ-representations" row gives +11.0% / +10.6% (LIVE-VQC), +5.8% / +5.7% (KoNViD-1k), +2.3% / +1.7% (CVD2014), +8.3% / +9.4% (LIVE-Qualcomm) and +7.7% / +8.7% (YouTube-UGC) tab. 5.

Ablation on fragments (SRCC / PLCC) tab. 6. The FANet structure is held fixed §4.5.

Ablation on FANet tab. 7.

Single-sample reliability tab. 8. The paper reports that a single sample is nearly as accurate as a 6-sample ensemble.

Robustness by resolution on LIVE-VQC (SRCC / PLCC / KRCC) tab. 9. Group sizes: (A) 1080P, 110 videos; (B) 720P, 316 videos; (C) ≤540P, 159 videos.

Dataset sizes tab. 1 §2 §4.1. LSVQ has 39,076 videos in total §2. The paper names 28,056 for training, 7,400 for LSVQ test and 3,600 for LSVQ 1080p §4.1. These sum to 39,056, so 20 videos are not accounted for (derived; not explained by the paper). The paper's dataset table also lists KoNViD-1k with 1,200 videos, LIVE-VQC with 585 and YouTube-UGC with 1,147 (an incomplete public release) tab. 1 §4.4.

Paper vs code

No code comparison was made. The authors' repository, timothyhtimothy/FAST-VQA, is linked from the abstract. Its LICENSE is the S-Lab License 1.0, which permits non-commercial use only [code_license]. wikillm publishes with advertising, so this card uses the paper alone and does not use or cite the repository's code. The code, code_commit and code_license fields record the repository for reference only. Anyone who wants the training recipe or the code-level choices should read the repository under the S-Lab terms.

Pitfalls

• The training recipe is missing. The optimiser, learning rate, schedule, epoch or iteration count, augmentation, and training hardware are not in the compiled paper §4.1. The LaTeX source keeps an appendix inside a comment block, so it is not part of the published paper. This card does not use it. An implementer must choose these values and say so.
• The sampling-offset distribution is undefined. The paper says "random" for the patch position inside a cell §3.1 eq. 2. It does not say whether the offset is uniform, whether it changes every iteration or epoch, or whether test-time inference uses one fixed offset or several.
• Test-time protocol is not stated for the headline numbers. The single-versus-ensemble comparison is reported only in the stability analysis §4.7 tab. 8. Whether the benchmark numbers in tab. 3 use one sample or an ensemble is not stated.
• Frame selection is undefined. The paper says the model uses T frames tab. 2 but does not say which frames are picked from a longer video, or with what stride.
• Grid edges are fractional. Equation 1 uses i·H/G_f, which is not an integer for most resolutions. For example, 1080/7 ≈ 154.3 (derived). The paper does not state the rounding rule §3.1 eq. 1.
• Small cells are not discussed. A cell must be at least S_f = 32 pixels in each direction for a raw patch to fit tab. 2 §3.1. The paper does not say what happens when a cell is smaller, for example in very small frames.
• Mini-patch boundaries and token boundaries. The gate G is defined per pair of tokens in the same mini-patch §3.2. The paper does not state the backbone's token size or whether mini-patch edges fall on token edges. An implementer should check that the gate is well defined at the chosen token size.
• Pool operator and regressor are not specified. IP-NLR pools after regression §3.2 eq. 7, but the pooling operator (mean, sum, max or other) and the regressor's layer count, width and activation are not stated. It is also not stated whether scores are per token or per mini-patch.
• Relative bias table is underspecified. T^real and T^pseudo are indexed by FRP(i, j), the relative position in the fragment §3.2. The table shape and the index mapping are not stated.
• Improvement percentages are mostly relative. The "+10%" over PVQ on 1080p is a relative gain. The absolute PLCC gain is 0.075 tab. 3 (derived). Read the percentages with that in mind.
• The running-time claim does not match the table. The text says FAST-VQA runs "up to 247×" faster than PVQ §4.3. At 1080P, the table gives 13.79 / 0.045 ≈ 306× on GPU and 538.4 / 9.142 ≈ 59× on CPU (derived). The basis for 247× is not stated. The timing hardware is not named.
• FLOPs are fixed by design, but accuracy is not. FAST-VQA's FLOPs are 279G at every resolution tab. 4. The per-sample input is always T × (G_f·S_f)², so cost does not grow with frame resolution. That is the source of the efficiency claim. Accuracy at 540P and 720P is reported separately tab. 4 tab. 9.
• Coverage is small. A single fragment keeps about 2.4% of the pixels of a 1080P frame: 7² × 32² / (1080 × 1920) ≈ 2.42% (derived, matching the paper's "2.4%" §4.7). Sample stability is high tab. 8, but a single sample still sees a small part of each frame.
• Small subsets in the robustness table. The 1080P group of LIVE-VQC has 110 videos §4.7 tab. 9. Differences of 0.01 between rows are within the noise that a 110-video subset allows. The paper does not report variance for that table.
• Transfer baseline is undefined. "FAST-VQA without VQ representations" tab. 5 is described as competitive, but the main text does not say how it was initialised.
• Fine-tuning split ratio is not stated. The paper says each dataset is split at random 10 times §4.4 but does not give the train/test ratio.
• Equation and table numbers are inferred. No compiled PDF was available. Equation numbers were counted from the LaTeX source order: eq. 1 to 4 for GMS, eq. 5 and 6 for GRPB, eq. 7 for IP-NLR. Table and figure numbers follow the same order. Check these against the PDF before citing them publicly.
• Mismatched labels in the source. The local quality map figure carries the LaTeX label fig:6 but is the seventh figure in order, so it is cited here as fig. 7.

Lineage

• Builds on: Video Swin Transformer (Tiny) as the backbone, pretrained on Kinetics-400 §3.2 §4.1. Named in the paper, with no card in this wiki.
• Compared against: BRISQUE, TLVQM, VIDEVAL, VSFA, PVQ (with and without patch labels), a full-resolution Swin-T with fixed features, RAPIQUE, GST-VQA, CoINVQ and CNN ensembles tab. 3 tab. 5.
• Superseded by: not known from wiki content.