Learning a No-Reference Quality Metric for Single-Image Super-Resolution

In plain words

Super-resolution algorithms turn a small, blurry picture into a larger, sharper one. Many exist, and the usual ways to grade them either need the original sharp picture, which is often unavailable in practice, or measure pixel differences, which do not match what people actually find pleasing.

The authors first asked many people to rate super-resolved images by eye. They then trained a scoring program on those ratings. The program looks only at the output image: it measures how its frequency content and small patches look statistically, and it has learned from the human ratings which of those measurements go with a good-looking result.

The scorer can be used on its own to compare methods, or to pick the best output of several algorithms for the same input.

TL;DR

• A learned, reference-free quality score for super-resolved images, trained on human absolute ratings §3 §4.
• Three feature groups (local DCT statistics, global wavelet statistics, and singular values of image patches) each feed a random forest, and a linear layer combines the three outputs §4.4 eq. 7.
• Author-reported: best Spearman correlation with human scores among the compared metrics, both under random 5-fold cross-validation and under the two harder held-out schemes tab. 7 tab. 8 tab. 9.
• Only the author-reported numbers are available; the code was not checked (see Paper vs code).

Problem

Evaluating super-resolution (SR) output is hard for three reasons, which the authors give as motivation §1 §2:

• Full-reference metrics (MSE, PSNR, SSIM, IFC) need a ground-truth image. That is often unavailable in practice §1.
• Those metrics were designed for generic signal fidelity, not human perception. The authors' example is an SR image shifted by half a pixel, which looks fine to people but scores poorly on PSNR, SSIM and IFC fig. 1 tab. 1.
• Existing no-reference image quality metrics are trained on noise, compression and fading distortions, not on SR artifacts such as incorrect high-frequency detail §2.3.

The method assumes the blur kernel width is known for each scaling factor, and it targets the large quality differences caused by the scaling factor §3.

Method

The paper has two parts: a human subject study that produces a dataset, and a learned no-reference metric trained on that dataset §1.

Dataset construction §3.

1. Take 30 high-resolution images from the BSD200 set, chosen by their PSNR spread so they cover a range of high-frequency detail §3.
2. Make low-resolution (LR) inputs with a Gaussian-blur downsampling model eq. 1. The noise term used in the authors' earlier benchmark is removed §3.
3. Choose one kernel width σ per scaling factor s tab. 2.
4. Super-resolve each LR input with 9 algorithms, giving 1,620 SR images (30 images × 6 scales × 9 algorithms) §5.
5. Collect 50 absolute ratings from 0 to 10 per image §3 fig. 4. The perceptual score is the mean of the median 40 ratings §3 §5.

Features §4. Each SR image is described by three groups, computed on pyramids to reduce sensitivity to scale §4:

1. Local frequency (18 values) §4.1 tab. 5. Take DCT coefficients of blocks, split them into three groups (DC excluded) fig. 10(a), and fit a generalized Gaussian distribution (GGD) to the coefficients eq. 2. Use the shape parameter γ, and the normalized deviation σ̄ = σ/μ per group with its variation Σ. Pool each over blocks by mean, and add the first and last 10% of pooled values §4.1.
2. Global frequency (45 values) §4.2 tab. 5. Apply a complex steerable pyramid §4.2. Wavelet coefficients are divisively normalized by a neighbourhood of N = 15 filters that share one mixer §4.2 fig. 10(b) eq. 3. Use the GGD shape γ of each band and of each orientation's concatenated bands, plus a structural correlation across scales and across bands §4.2.
3. Spatial discontinuity (75 values) §4.3 tab. 5. Apply PCA to 5×5 patches on a three-level pyramid and use the singular values as features §4.3 §5.1.

Regression §4.4 alg. 1.

• Train one regression forest per feature group. Node splits maximize a log-determinant objective built on a conditional covariance from probabilistic linear fitting eq. 4 eq. 5. Each forest prediction is the mean over its T trees eq. 6.
• Combine the three forest outputs with one linear layer whose weights λ are fitted by least squares eq. 7.

# Predict a quality score for one SR image  [§4] [alg. 1]
# Input:  sr_image (H, W) grayscale
# Output: scalar quality score (0 worst, 10 best)

def predict_quality(sr_image, forests, weights):
    # Three feature groups from the image  [§4.1] [§4.2] [§4.3]
    local = local_frequency_features(sr_image)   # 18 values  [tab. 5]
    wavelet = global_frequency_features(sr_image)  # 45 values  [tab. 5]
    spatial = spatial_features(sr_image)         # 75 values  [tab. 5]

    # One forest per group, averaged over its T trees  [§4.4] [eq. 6]
    scores = []
    for group, forest in zip([local, wavelet, spatial], forests):
        scores.append(forest.predict_mean(group))

    # Linear combination of the three forest outputs  [§4.4] [eq. 7]
    return sum(w * s for w, s in zip(weights, scores))

Hyperparameters

Values are as stated in the paper. Where the paper gives no value, it is marked not stated.

Results

Author-reported. Setup: 1,620 SR images from 30 reference images and 9 SR algorithms, with 50 ratings each §5. Baselines are retrained on the same dataset except DIVINE, BIQI, and the original-model rows in tab. 10. Metric is the Spearman rank correlation with the human scores (higher is better).

Headline table, 5-fold cross-validation with metrics retrained on the SR dataset tab. 7. The first column is the proposed metric. The next six are no-reference baselines, then one semi-reference baseline, then four full-reference metrics.

Other validation results tab. 6 tab. 8 tab. 9 tab. 10, overall rows only:

tab. 10 (original, non-retrained baselines): BRISQUE 0.644, BLIINDS 0.763, CORNIA 0.809, CNNIQA 0.833 overall. Per-method rows in tab. 6, tab. 8, tab. 9 and tab. 10 were not transcribed here; see the paper.

Reported error fig. 11: root-mean-square error between predicted and human score is 0.87 in 5-fold CV and below 1.4 in the two leave-out schemes, on a 0–10 scale.

Run time in seconds, averaged, as reported §5.3: ours 13.31; BRISQUE 0.14; BLIINDS 23.57; DIVINE 9.51; BIQI 1.21; CORNIA 3.02; CNNIQA 12.68; NSSA 0.28; IFC 0.61; SSIM 0.13; FSIM 0.18; PSNR 0.02.

Paper vs code

Code not checked. The paper says the source code and dataset are at the authors' project page §5. That page is a Google Sites URL, not a repository, and this compile run could not open it, so no code, code_commit or code_license is recorded. Searching GitHub was not possible in this run, so no repository was found either. Re-run the code step (project page, then GitHub search) before relying on any statement about the implementation.

Known gaps between the paper's text and its own tables, to verify against the code:

• The steerable-pyramid parameters are written as "α and θ to be 2 and 6" §5.1, but the text then describes 12 subbands with α ∈ {1, 2} and six orientations. The 6 in θ is a count of orientations, not an angle.
• Table 5 lists six values per local-frequency feature, but the text does not derive the counts (3 pyramid levels, 7×7 blocks, first/last 10%) tab. 5 §4.1.
• The 18 global-frequency γ values tab. 5 are described as "each band and concatenated bands" §4.2; the 12 across-scale and 15 across-band correlation counts are not derived in the text.

Pitfalls

• Subject score outliers. "Mean of the median 40 of 50 scores" is the only rule given. How the 10 discarded ratings are picked (by distance from the median, or by subject) is not stated §3.
• Forest and linear-layer leakage. The weights λ are fitted on the forest outputs ŷₙ eq. 7, which come from forests trained on the same training data. The paper does not say whether these outputs are out-of-fold. Using in-sample forest predictions to fit λ inflates training fit.
• Tree count chosen on what data? T = 2000 was selected by sweeping 100–5000 §5.1. The paper does not say whether the sweep used a held-out split separate from the reported test folds.
• GGD density without scale. The density in eq. 2 has no scale parameter and is normalized only for unit scale. How the authors handle scale when fitting, and whether eq. 2 is the exact form used, is not stated.
• Structural correlation. The constant c₀ is not stated §4.2. The text calls σ_x and σ_y "windowed variances" but the formula is written in the SSIM-like form; check the code for the exact definition.
• Divisive normalization and mixer. The mixer ẑ = √(YᵀQ⁻¹Y/N) and the covariance Q are not estimated in any detail §4.2 eq. 3. The complex (real plus imaginary) decomposition is a deliberate departure from the cited prior work §4.2.
• Forest settings. Depth, minimum leaf size, and the random subset size 𝒯ⱼ are not stated §4.4. The objective depends on a probabilistic linear-fit covariance that is only described in [MSR-TR-2011-114].
• Stochastic evaluation. Results are the mean of 100 runs §5.2. Single-run numbers may differ.
• Scope of the data. Only 30 reference images and a fixed set of scaling factors with fixed kernel widths tab. 2 are covered. The paper states the kernel width is assumed known §3. Behaviour on other blur models or unseen scales is not evaluated.
• Cost. The reported runtime is 13.31 seconds per image in MATLAB §5.3, against 0.13 s for SSIM and 0.02 s for PSNR. The image size used for the timing is not stated.
• Perception-guided fusion. Patch fusion uses graph cut and Poisson blending §6. The paper gives no graph-cut energy or parameters. The paper also reports these outputs only as examples fig. 17 fig. 18, without a quantitative fusion study.
• Known typo. A table header reads "SRV-DCT" for SVR-DCT tab. 6, and a text reference to "Table 1 in the manuscript" is left over from a draft §5.3.

Lineage

• Builds on: the authors' SR benchmark work [Yang14_ECCV], which uses full-reference metrics and a noisy downsampling model. This paper removes the noise term §3.
• Builds on: the Yeganeh et al. spatial statistics approach (NSSA) for spatial features §4.3, and the generalized Gaussian and divisive-normalization features from BRISQUE, BLIINDS and DIVINE-style IQA work §4.1 §4.2.
• Compared against: full-reference PSNR, SSIM, IFC and FSIM; no-reference BRISQUE, BLIINDS, DIVINE, BIQI, CORNIA and CNNIQA; semi-reference NSSA §5.
• Superseded by: not stated in this card.