Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective

In plain words

Image quality models try to guess how good a photo looks to a person, usually without a clean original to compare it with. They have many parameters but only a small number of human quality scores to learn from.

This paper builds on CLIP, a model that learned to match pictures with text. Instead of predicting one number directly, the model checks how well an image fits many short sentences such as "a photo of a city with blur artifacts, which is of good quality". Each sentence bundles three things: how good the photo is, what kind of scene it shows, and what kind of damage it has. The model learns all three at once, and the quality answer is read off the same match scores. Training weights the three goals automatically, so no one has to tune them by hand. The authors report that the extra scene and damage labels help the quality score, and that one set of weights works across several quality datasets.

TL;DR

LIQE is a no-reference image quality model built by fine-tuning CLIP ViT-B/32 §4.1. It scores each image against 495 joint text labels (5 quality levels, 9 scenes, 11 distortion types) and reads quality, scene and distortion from the same softmax §3.2 eq. 3. The authors report median SRCC of 0.970 on LIVE and 0.919 on KonIQ-10k over ten random splits tab. 1. The numbers are author-reported, and several training details differ between the paper and the official code (see Paper vs code).

Problem

Blind image quality assessment (BIQA) predicts perceived quality with no reference image §1. Deep BIQA models have many parameters but few human quality annotations (mean opinion scores, MOS) §1. Training tricks that rely on synthetic distortions or pseudo-labels assume that pristine references exist and that distortions can be simulated, and these assumptions do not hold for realistic camera distortion §1. Multi-dataset joint training is one route. The paper asks whether auxiliary tasks, such as scene classification and distortion identification, can help BIQA in a multitask setting §1. Scene classification looks conceptually at odds with quality (distortion-insensitive versus distortion-sensitive features), while distortion identification looks related §1 fig. 1.

Method

The model is one CLIP image encoder and one CLIP text encoder, both trained end to end. All parameters of the image encoder are shared across the three tasks §2.3.

1. Label space. Quality is a Likert scale of five levels, bad, poor, fair, good and perfect. The predicted quality is the expected level under the marginal distribution §3.1 eq. 1. There are nine scene categories, where an image may carry several labels, and eleven dominant distortion types including "others", which covers pristine images §3.1. Each combination becomes the text "a photo of a(n) {scene} with {distortion} artifacts, which is of {quality} quality", giving 5 x 9 x 11 = 495 candidates §3.1.
2. Visual embedding. The paper crops U sub-images of 224 x 224 pixels without resizing, because resizing can change perceived quality §3.2. The cosine similarity of each crop with each text embedding is averaged over the U crops §3.2 eq. 2. Texts are lower-cased byte-pair encoded §3.2.
3. Joint distribution. A softmax over all 495 scores with a temperature tau_1 gives the joint probability. The paper calls tau_1 learnable §3.2 eq. 3.
4. Marginals. Summing the joint over the other two tasks gives the quality, scene and distortion marginals §3.2.
5. Quality loss. For each pair of images from the same dataset, the label is 1 if the first image has a MOS at least as high as the second, and 0 otherwise §3.2 eq. 4. The model probability is a Thurstone model with unit variance §3.2 eq. 5. The loss is the fidelity loss §3.2 eq. 6.
6. Scene loss. A multi-label fidelity loss averaged over the nine scene categories §3.2 eq. 7. A softmax variant was also tried with similar results §3.2 eq. 8.
7. Distortion loss. A multi-class fidelity loss over the eleven distortion types, using the marginal of the joint §3.2 eq. 9.
8. Total loss. The quality term averages over pairs, and the scene and distortion terms average over the batch. Each term has its own weight lambda(t) §3.2 eq. 10.
9. Loss weights. Dynamic weight average: each weight is a softmax over the relative descent rate of the three losses, with a fixed temperature tau_2 §3.2 eq. 11. The paper does not give tau_2.
10. Multi-dataset training. One mini-batch per dataset is sampled per iteration, and the pairs come only from within a dataset §3.2 §4.1.

# Training step for LIQE  [§3.2] [eq. 2-10]
# Input:  one mini-batch per IQA dataset; each sample has U crops
# Output: scalar loss

def training_step(batches, text_embeds, weights):
    # Stack the samples of all datasets into one batch  [§4.1]
    crops = concatenate_over_datasets(batches)     # (N, U, 3, 224, 224)
    # Cosine similarity of each crop with the 495 texts, averaged  [eq. 2]
    logits = mean_over_crops(cosine(crops, text_embeds))  # (N, 495)
    # Softmax over the 5 x 9 x 11 joint labels  [eq. 3]
    joint = softmax(logits / tau_1)  # tau_1 is learnable in the paper
    # Marginalise the joint over the other two tasks  [§3.2]
    p_quality = sum_over_scene_and_distortion(joint)  # (N, 5)
    p_scene = sum_over_quality_and_distortion(joint)  # (N, 9)
    p_distortion = sum_over_quality_and_scene(joint)  # (N, 11)
    # Quality score is the expected level, 1 to 5  [eq. 1]
    quality = p_quality @ [1, 2, 3, 4, 5]
    # Quality loss over same-dataset pairs of the batch  [eq. 4-6] [eq. 10]
    pairs = same_dataset_pairs(batches)
    loss_quality = mean(pairwise_fidelity(quality, pair) for pair in pairs)
    # Scene loss: mean of binary fidelity over the 9 categories  [eq. 7]
    loss_scene = mean(binary_fidelity(p_scene, scene_targets))
    # Distortion loss: multi-class fidelity over the 11 types  [eq. 9]
    loss_distortion = 1 - sum(sqrt(p_distortion * distortion_targets))
    # Weighted sum of the three losses  [eq. 10]
    losses = [loss_quality, loss_scene, loss_distortion]
    return dot(weights, losses)

# Pairwise fidelity loss for two images  [eq. 4-6]
# Input:  predicted quality and MOS of images i and j
# Output: scalar loss

def pairwise_fidelity(quality_i, quality_j, mos_i, mos_j):
    # Label is 1 if image i is at least as good as image j  [eq. 4]
    target = 1 if mos_i >= mos_j else 0
    # Thurstone probability with unit variance  [eq. 5]
    predicted = Phi((quality_i - quality_j) / sqrt(2))
    # Fidelity loss  [eq. 6]
    loss = 1 - sqrt(target * predicted)
    loss = loss - sqrt((1 - target) * (1 - predicted))
    return loss

# Loss weights from the descent rate of each task  [eq. 11]
# Input:  mean loss per task in the last two epochs, temperature tau_2
# Output: one weight per task

def dwa_weights(losses_last, losses_before, tau_2):
    # Ratio of each task's mean loss over the last two epochs  [eq. 11]
    descent = losses_last / losses_before
    # Softmax over tasks at temperature tau_2  [eq. 11]
    weights = softmax(descent / tau_2)
    return weights

Key equations, as in the paper:

\hat{q}(\bm{x}) = \sum_{c=1}^{5} \hat{p}(c \mid \bm{x})\, c eq. 1

\mathrm{logit}(c,s,d \mid \bm{x}) = \frac{1}{U}\sum_{u=1}^{U} \cos\!\big(\bm{F}_{u\bullet}(\bm{x}),\, \bm{G}_{v\bullet}\big) eq. 2

\hat{p}(c,s,d \mid \bm{x}) = \frac{\exp\big(\mathrm{logit}(c,s,d \mid \bm{x})/\tau_1\big)}{\sum_{c,s,d}\exp\big(\mathrm{logit}(c,s,d \mid \bm{x})/\tau_1\big)} eq. 3

\hat{p}(\bm{x},\bm{y}) = \Phi\!\left(\frac{\hat{q}(\bm{x}) - \hat{q}(\bm{y})}{\sqrt{2}}\right) eq. 5

\ell_q(\bm{x},\bm{y}) = 1 - \sqrt{p(\bm{x},\bm{y})\,\hat{p}(\bm{x},\bm{y})} - \sqrt{\big(1-p(\bm{x},\bm{y})\big)\big(1-\hat{p}(\bm{x},\bm{y})\big)} eq. 6

\ell_s(\bm{x}) = \frac{1}{|\mathcal{S}|}\sum_{s\in\mathcal{S}} \Big(1 - \sqrt{p(s\mid\bm{x})\,\hat{p}(s\mid\bm{x})} - \sqrt{\big(1-p(s\mid\bm{x})\big)\big(1-\hat{p}(s\mid\bm{x})\big)}\Big) eq. 7

\ell_d(\bm{x}) = 1 - \sum_{d\in\mathcal{D}} \sqrt{p(d\mid\bm{x})\,\hat{p}(d\mid\bm{x})} eq. 9

\ell(\mathcal{B},t) = \frac{1}{|\mathcal{P}|}\sum_{(\bm{x},\bm{y})\in\mathcal{P}} \lambda_q(t)\,\ell_q(\bm{x},\bm{y}) + \frac{1}{|\mathcal{B}|}\sum_{\bm{x}\in\mathcal{B}} \big(\lambda_s(t)\,\ell_s(\bm{x}) + \lambda_d(t)\,\ell_d(\bm{x})\big) eq. 10

\lambda_j(t) = \frac{\exp\big(w_j(t-1)/\tau_2\big)}{\sum_i \exp\big(w_i(t-1)/\tau_2\big)}, \qquad w_j(t-1) = \frac{\ell_j(t-1)}{\ell_j(t-2)} eq. 11

Figures: fig. 1 shows why scene and distortion can conflict: a blurred parrot photo is easy for humans, but blur is what a quality model must detect §1. fig. 2 is the system diagram §3. fig. 3 shows gMAD results against UNIQUE §4.4. No figures are reproduced here.

Hyperparameters

Values in the paper and in the official training script. Where they differ, both are given; see Paper vs code.

Results

Author-reported. Setup: six IQA datasets, 70/10/20 random splits repeated ten times, median over the ten sessions §4.1 tab. 1. Baselines: three opinion-unaware methods and seven deep methods §4.2. The table reports the median and the standard deviation (in brackets) over ten sessions tab. 1. The model selection rule is not stated in the paper (see Paper vs code).

Table 1: SRCC (median, SD in brackets) tab. 1

Table 1: PLCC (median, SD in brackets) tab. 1

On PLCC, LIQE is not the top method on LIVE, CSIQ or KonIQ-10k in this table (for example, HyperIQA 0.968 and DBCNN 0.966 on LIVE) tab. 1. The paper's bold marks follow its own table.

Table 2: SRCC under the cross-dataset setup (models trained on KADID-10k and KonIQ-10k as in the subscripts) tab. 2

The paper states that none of the methods performs well on PIPAL §4.3 tab. 2.

Table 3: Median SRCC in the multi-dataset realignment experiment tab. 3

Table 4: Ablation, median SRCC over ten sessions tab. 4

The gap between LIQE and equal weighting is 0.002 in mean SRCC tab. 4. The paper reports no spread for the ablation.

Table 5: Task combinations, mean over six datasets tab. 5

The paper's realignment check uses 150 sampled images and six monotone mappings §4.4.

Paper vs code

Code: zwx8981/LIQE at a64655f7b84b9543b513394472851884cde5b257. The paper links it from the abstract, and its owner is the first author, Weixia Zhang. MIT licence, copyright Weixia Zhang. The paper's main results come from train_unique_clip_weight.py, and the evaluation from BIQA_benchmark.py.

• Joint temperature. The paper treats tau_1 as learnable §3.2 eq. 3. The code freezes CLIP's logit_scale for every option [code: train_unique_clip_weight.py#L129] and calls freeze_model(opt) with opt=0 train_unique_clip_weight.py. The effective scale is fixed at CLIP's pretrained value.
• DWA weights. Eq. 11 has no factor for the number of tasks §3.2 eq. 11. The code multiplies each softmax weight by n_tasks, so the three weights sum to 3 weight_methods.py. The first two epochs use weight 1 weight_methods.py, which the paper does not mention. The loss averages used in the ratio are epoch means of per-step losses [code: weight_methods.py#L185-L186, #L192-L193].
• DWA temperature and alpha. The paper gives no tau_2 value §3.2. The code uses temp = 2.0 train_unique_clip_weight.py and passes alpha = 1.5, which the DWA class does not use weight_methods.py.
• Tied MOS in pair labels. Eq. 4 gives label 1 when q(x) >= q(y) §3.2 eq. 4. The code sets the label to 0.5 for ties (0.5 * (sign + 1)) [code: MNL_Loss.py#L151].
• Pair averaging. Eq. 10 averages over all pairs of the union §3.2 [eq. 10]. The code averages the pairs within each dataset, then divides by the number of datasets MNL_Loss.py. A 4-image dataset gives 6 pairs, a 16-image dataset gives 120. In the code a pair from a 4-image dataset therefore carries weight 1/36 and a pair from a 16-image dataset weight 1/720, against 1/264 under eq. 10 for the batch in this setup.
• Scene loss form. The paper also tries a softmax form §3.2 eq. 8. The code only has the binary multi-label form averaged over nine categories [code: MNL_Loss.py#L42-L59].
• Text encoder. The paper names GPT-2 with 63M parameters as the text encoder §4.1. The code calls clip.load("ViT-B/32") and so uses CLIP's own text transformer train_unique_clip_weight.py.
• Label names. The paper writes "indoor scene", "night scene" and "still-life" §3.1. The code uses "indoor", "night" and "still_life" [code: train_unique_clip_weight.py#L19-L26] train_unique_clip_weight.py. The paper's "spatially-localized" is "spatial" in the code [code: train_unique_clip_weight.py#L35-L46].
• Distortion labels. The paper gives only the eleven classes §3.1. The mapping from the 33 dataset-specific labels to those classes is in the code train_unique_clip_weight.py.
• Resizing. The paper avoids resizing, because it changes perceived quality §3.2. The code resizes images whose two sides are both at least 768 pixels so that the shorter side is 768 utils.py utils.py. Smaller images are not resized.
• Training crops. The paper says "randomly crop" §4.1. The code takes crops from a grid with stride 32, or stride 48 when one side is at least 1024 pixels, and draws the indices with replacement ImageDataset2.py [code: ImageDataset2.py#L88].
• Training flip. The training preprocessing adds a random horizontal flip utils.py, which the paper does not mention.
• Test crops. The paper says "randomly crop" 15 sub-images at inference §4.1. The evaluation code takes 15 evenly spaced crops [code: ImageDataset2.py#L81-L86].
• Stand-alone inference. LIQE.py always uses stride 32, never resizes, and takes 15 evenly spaced crops LIQE.py LIQE.py [code: LIQE.py#L72-L77]. The benchmark pipeline resizes and uses stride 48 for large images utils.py ImageDataset2.py. Scores from the two paths need not match.
• Epoch. The paper uses 80 epochs §4.1. In the code an epoch is 200 iterations; each iteration draws one mini-batch from each of the six loaders, 48 images in total train_unique_clip_weight.py [code: train_unique_clip_weight.py#L183-L197].
• Learning-rate schedule. Cosine annealing with T_max = 5 is stepped once per epoch train_unique_clip_weight.py train_unique_clip_weight.py. The script also steps the scheduler again when the learning rate is exactly zero train_unique_clip_weight.py. Under T_max = 5 the cosine reaches zero at epoch 5 and rises again. This is inferred from the code and was not run.
• Model selection. The paper does not say which checkpoint the table uses. The code picks the epoch by the average validation SRCC over the six datasets and saves quality_best_ckpt.pt train_unique_clip_weight.py train_unique_clip_weight.py [code: train_unique_clip_weight.py#L417-L448]. The benchmark loads that checkpoint BIQA_benchmark.py.
• Median or mean. The paper reports median SRCC over ten sessions tab. 1. The code's final_avg calls np.mean and labels the result "median" [code: BIQA_benchmark.py#L377-L385].
• PLCC. The code's PLCC and RMSE are computed after a four-parameter logistic fit to the test set BIQA_benchmark.py. The paper does not state this for its main table.
• Mixed precision. On CUDA the code casts weights to fp32 for the optimiser step, then calls clip.model.convert_weights to cast them back [code: train_unique_clip_weight.py#L303-L308]. The CLIP package is not in the repo, so the effect of that call was not checked here.
• Checkpoint format. The training script saves a dict with model_state_dict train_unique_clip_weight.py. LIQE.py loads the file as a raw state dict LIQE.py. The benchmark loads the dict form BIQA_benchmark.py.
• Logged distortion accuracy. In the test record for KADID-10k the distortion entry uses the validation value dist_acc6 train_unique_clip_weight.py. The model weights are unaffected, but the all_results dict saved in each checkpoint carries the wrong value train_unique_clip_weight.py.
• Fidelity epsilon. The code adds 1e-8 inside each square root [code: MNL_Loss.py#L9] MNL_Loss.py. The paper writes no epsilon §3.2.
• Pretrained weights and data. The README links the pretrained weights on Google Drive and Baidu README.md. The IQA image folders are not in the repository, only the split files train_unique_clip_weight.py.
• Third-party re-training. The README points to IQA-PyTorch liqe (re-trained on KonIQ-10k) and liqe_mix (multi-dataset) README.md. Neither is the paper's model.

Pitfalls

• Text tower cost. The text encoder is trained, so all 495 prompts go through it on every training step train_unique_clip_weight.py train_unique_clip_weight.py. This is a per-step cost that the abstract does not mention.
• Crops per image. Training uses 3 crops and testing uses 15 §4.1. The cost of inference is 15 forward passes per image, each with all 495 texts scored.
• Pair weights. A 4-image dataset and a 16-image dataset weigh their pairs very differently under the code's averaging (see Paper vs code).
• Numerical precision. Updates are cast back to half precision after each step on CUDA train_unique_clip_weight.py. With a learning rate of 5e-6, small updates may round away. Verify this in your setup before copying the schedule.
• Scene and distortion labels per image. The training loader sets every valid scene label to 1, so multi-label targets are used [code: train_unique_clip_weight.py#L210-L219]. The paper's binary form is the same §3.2. The per-image labels sit in the split files shipped with the repo, but the images they point to are not in the repo.
• Reported spread. The table reports medians and SDs over ten sessions [tab. 1]. The selection rule is not stated in the paper, so the reported epoch cannot be reproduced exactly from the text.
• Ablation margins. Several ablation differences are small (for example 0.920 against 0.922 mean SRCC) and the paper reports no per-variant spread tab. 4.
• Unstated constants. tau_2, the cosine T_max and the seed are not in the paper. The code's values are listed under Hyperparameters.
• Intermediate checks. The validation SRCC is computed with the same 15 crops used at test time train_unique_clip_weight.py. Use this to check your run, not as an independent test score.

Lineage

• Builds on: CLIP §2.2 §3.2. Named in the paper, with no arXiv id in the source, so not linked here.
• Builds on: UNIQUE's joint training with pairwise learning-to-rank over several datasets §3.2 §4.5. Named in the paper, no card here.
• Closest prior work: Wang et al., who use CLIP for quality through prompt engineering with two quality levels §2.2 §4.5.
• Compared against: NIQE, ILNIQE, Ma19, PaQ2PiQ, KonCept, MUSIQ, DBCNN, HyperIQA, TreS and UNIQUE §4.2.
• Loss weighting: dynamic weight average (Liu et al.) §2.3 eq. 11.