Image quality assessment

Overview

Image quality assessment (IQA) gives a score to an image that predicts how good people will find it. The classic setting compares the image with an undistorted reference, which the paper calls full-reference. The other setting has no reference and must judge the image alone 1612.05890 §1 §2. Blind (no-reference) IQA is the setting of 2303.14968 §1.

The first paper's motivating case is single-image super-resolution (SR), where a small image is enlarged. The authors argue that full-reference measures such as MSE, PSNR and SSIM need the high-resolution ground truth, which is often not available, and that they match human perception poorly on SR outputs 1612.05890 §1 tab. 1. The second paper's motivating case is realistic camera distortion, where it argues that training tricks based on synthetic distortions or pseudo-labels do not carry over 2303.14968 §1.

The invariant shared by the variants below: the score is fitted or designed to track human ratings of the same kind of images. The first card trains a regressor on 81,000 human ratings 1612.05890 §3 tab. 4. The second trains on pairs of mean opinion scores from the same dataset with a fidelity loss 2303.14968 §3.2 eq. 4 eq. 6.

Variants

Two no-reference cards are available. The first two subsections describe them. The remaining subsections compare with the baselines that the first paper reports, as the paper reports them; those baseline numbers are not checked here.

Learned no-reference metric from human ratings

The first paper trains a regressor to predict the mean human score of each SR image from low-level image statistics, without the original image. The statistics come from three groups: local DCT block statistics, global steerable-pyramid statistics and spatial singular values of 5×5 patches 1612.05890 §4.1 §4.2 §4.3. Each group feeds a random forest, and the three forest outputs are combined linearly with least-squares weights 1612.05890 §4.4. Card: Learning a No-Reference Quality Metric for Single-Image Super-Resolution. Author-reported Spearman correlation with human scores is 0.931 under 5-fold cross-validation 1612.05890 tab. 7.

Vision-language no-reference model (LIQE)

LIQE fine-tunes CLIP ViT-B/32 end to end 2303.14968 §4.1. It scores each image against 495 short text labels, one for each combination of 5 quality levels, 9 scene categories and 11 distortion types 2303.14968 §3.1. The quality score is the expected level under the quality marginal of the same softmax that gives the scene and distortion labels 2303.14968 §3.2 eq. 1 eq. 3. Training combines a pairwise fidelity loss on quality with scene and distortion losses, and sets their weights with dynamic weight average 2303.14968 §3.2 eq. 10 eq. 11. The model is trained jointly on six datasets, with 70/10/20 random splits repeated ten times 2303.14968 §4.1. Card: Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective.

Author-reported median SRCC over the ten splits: 0.970 on LIVE, 0.936 on CSIQ, 0.930 on KADID-10k, 0.875 on BID, 0.904 on CLIVE and 0.919 on KonIQ-10k 2303.14968 tab. 1. On the cross-dataset setup it reports 0.811 on TID2013, 0.881 on SPAQ and 0.478 on PIPAL, and says none of the compared methods performs well on PIPAL 2303.14968 §4.3 tab. 2.

The paper's ablation favours the fine-tuned encoder: a pretrained CLIP with two quality levels averages 0.693 mean SRCC, against 0.890 for the fine-tuned version with the same two levels 2303.14968 tab. 4. Adding the scene and distortion tasks to quality moves the mean SRCC from 0.915 to 0.922 2303.14968 tab. 5. These margins are small, and the paper reports no spread for its ablations 2303.14968 tab. 4. The official code differs from the paper in several training details, one of them the temperature, which the paper treats as learnable; see the card's Paper vs code section 2303.14968 §3.2.

Full-reference measures (baselines)

The first paper compares its metric with IFC, SSIM, FSIM and PSNR, which all use the ground-truth image 1612.05890 §5.2. Overall Spearman under 5-fold cross-validation is 0.810 for IFC, 0.635 for SSIM, 0.747 for FSIM and 0.604 for PSNR 1612.05890 tab. 7. The authors note that these are not a like-for-like comparison, since they see the reference and the proposed metric does not 1612.05890 §5.2. SSIM itself is covered on its own page: Structural similarity.

No-reference baselines

The first paper also reports six no-reference methods: BRISQUE, BLIINDS, DIVINE, BIQI, CORNIA and CNNIQA 1612.05890 §5.2. Their overall 5-fold Spearman is 0.802, 0.853, 0.589, 0.482, 0.919 and 0.904 respectively 1612.05890 tab. 7. The paper says DIVINE and BIQI are reported with their original models, while the others are retrained on the paper's data 1612.05890 §5.2. The paper's own text is inconsistent on this point; see the card's Pitfalls. The authors' method is ahead of CORNIA (0.919) and CNNIQA (0.904) on the main protocol. On leave-6-reference-images-out the margin over CORNIA is narrow, 0.852 against 0.843 1612.05890 tab. 7 tab. 8.

Semi-reference baseline

NSSA is the one semi-reference method in the first paper's comparison 1612.05890 §5.2. It scores 0.076 overall under 5-fold cross-validation 1612.05890 tab. 7. The card records that its learned data file is not public and that its rebuilding is not described, so this number is weakly supported 1612.05890 §5.2 tab. 7.

When to use which

The first paper gives no evidence that its learned metric beats the baselines on degradations other than the one it was trained on 1612.05890 §3 §5. The second paper gives no runtime and no SR-only result 2303.14968 §4.

Implementation sketch

Two sketches follow. The first is the scoring path of the learned metric of 1612.05890 §4; the second is LIQE inference 2303.14968 §3.2. Both are our own versions, not the papers' code. Feature functions are described in the first card's Method section and are not spelled out here.

# Score one image with the learned no-reference metric  [arxiv:1612.05890 §4]
# Input:  image (H, W) grey or (H, W, 3); trained forests and linear weights
# Output: scalar on the 0-to-10 scale of the training ratings  [arxiv:1612.05890 §3]
def score_no_reference(image, forests, weights):
    # Three feature groups, each computed on the image alone  [§4.1] [§4.2] [§4.3]
    features = [
        local_dct_statistics(image),
        global_steerable_statistics(image),
        spatial_singular_values(image),
    ]
    # One regression forest per group, averaged over its trees  [§4.4] [eq. 6]
    partial = [forest.predict(f) for forest, f in zip(forests, features)]
    # Linear combination of the three forest outputs  [§4.4] [eq. 7]
    return dot(weights, partial)

# Score one image with LIQE at inference  [arxiv:2303.14968 §3.2]
# Input:  image (H, W, 3); fine-tuned image encoder; 495 text embeddings
# Output: scalar quality on the 1-to-5 scale of the quality levels  [eq. 1]
def score_liqe(image, image_encoder, text_embeds, tau_1):
    # Sub-images of 224 x 224, cropped without resizing; 15 at test  [§3.2] [§4.1]
    crops = crop_224_without_resizing(image, count=15)
    # Cosine of each crop with each text, averaged over crops  [eq. 2]
    logits = mean(cosine(image_encoder(crops), text_embeds), axis=crops)
    # Softmax over the 5 x 9 x 11 joint labels  [eq. 3]
    p_joint = softmax(logits / tau_1)  # (5, 9, 11)
    # Marginal over the quality levels only  [§3.2]
    p_quality = sum_over_scene_and_distortion(p_joint)  # (5,)
    # Expected quality level  [eq. 1]
    return dot(p_quality, [1, 2, 3, 4, 5])

Contradictions

Across the two papers there is no direct numerical conflict, because they use different data and protocols. The first paper scores its own SR set of 1,620 images with 5-fold cross-validation 1612.05890 §5 tab. 7. LIQE uses six IQA datasets with ten random 70/10/20 splits 2303.14968 §4.1 tab. 1. The two sets of numbers should not be compared directly. LIQE's baselines do not include the first paper's metric 2303.14968 §4.2.

Inside the first paper, the caption of its Table 10 says the baseline columns are not retrained, yet three of the four equal the values in its Table 9 1612.05890 tab. 9 tab. 10. The card treats those non-retrained baseline numbers as unverified.

Papers

• Learning a No-Reference Quality Metric for Single-Image Super-Resolution: a no-reference metric for SR outputs, trained on 81,000 human ratings of 1,620 SR images; overall 5-fold Spearman 0.931 as reported by the authors.
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective: LIQE, a CLIP-based no-reference model that scores images against 495 joint quality, scene and distortion labels; median SRCC 0.970 on LIVE and 0.919 on KonIQ-10k as reported by the authors.