Image enhancement

Overview

The task is to take a photo whose light or colour is wrong and produce a version that looks correct. The light cases are low light, backlight, over- or under-exposure and uneven lighting 2205.14871 §1 2303.17569 §1. The colour case is underwater: water absorbs and scatters light, which causes colour casts, colour artefacts and blur 2111.11843 §1. Underwater images are often bluish or greenish, low in brightness and hazy 2103.07138 §IV-C. In all four variants the fix is a learned mapping from the input to a corrected image. The papers differ on where the correction comes from and what it is made of.

Variants differ in two choices: where the correction comes from (paired ground-truth photos, or a frozen vision-language model), and what the correction is made of (per-pixel affine terms with a global colour matrix and gamma, a single illumination map that the image is divided by, a full image-to-image generator, or curves in two colour spaces blended per pixel).

Variants

Per-pixel and global adjustment with paired training data

Predicts a per-pixel multiply and add, a 3x3 colour matrix and a gamma for each image, and is trained with a supervised loss on paired photos 2205.14871 §3.1 eq. 4. The local branch works at full resolution with depth-wise convolutions, and the global branch uses ten learned queries over a low-resolution encoding 2205.14871 §3.2 fig. 3. The whole model has about 0.09M parameters, and the authors report about 0.004 s per image 2205.14871 tab. 1 tab. 2. Card: IAT.

Author-reported results: PSNR 23.50 and SSIM 0.824 on LOL-V2-real 2205.14871 tab. 1; PSNR 25.32 and SSIM 0.920 on MIT-Adobe FiveK, expert C targets 2205.14871 tab. 2; average PSNR 20.34 on the exposure-correction set, which has five expert targets 2205.14871 tab. 3.

Unsupervised illumination map guided by CLIP

Trains an enhancer with no paired targets. A frozen CLIP model scores outputs against a positive and a negative prompt, and both prompts are learned 2303.17569 §3.1. The enhancer predicts a one-channel illumination map, and the output is the input divided by that map 2303.17569 §3.1. An identity loss keeps the output close to the input's content [arxiv:2303.17569 §3.1]. Training alternates prompt refinement, which uses a margin ranking loss, with enhancer tuning 2303.17569 §3.2. Card: CLIP-LIT.

Author-reported results on the BAID backlit test set (368 images, photographer retouched references): PSNR 21.579, SSIM 0.883, LPIPS 0.159 and MUSIQ 55.682, against 19.740 PSNR for the best released baseline, Zero-DCE [arxiv:2303.17569 tab. 1]. Replacing the learned prompts with fixed words drops PSNR to 14.748 2303.17569 tab. 3. Training uses 380 backlit and 384 well-lit images, unpaired 2303.17569 §4.

Paired transformer GAN for underwater colour

A U-shaped generator trained as a GAN on pairs of underwater photos and reference images. The skip connections are replaced by a channel-wise transformer, and a spatial transformer sits at the bottleneck 2111.11843 §3.2.1 2111.11843 §3.2.2 2111.11843 §3.2.3. The loss adds colour terms in LAB and LCH spaces to an RGB term, a perceptual term and the adversarial term 2111.11843 §3.3 eq. 13. Training data come from LSUI, which the authors built from 18 existing enhancers, image-quality scores and volunteer votes 2111.11843 §3.1.

Author-reported results: PSNR 24.16 and SSIM 0.93 on Test-L400, and PSNR 22.91 and SSIM 0.91 on Test-U90, the best of the ten compared methods 2111.11843 tab. 3. In the ablation, the pix2pix baseline reaches PSNR 19.34 on Test-L400 and adding only the channel transformer reaches 22.47 2111.11843 tab. 6. Cost at 256 × 256: 65.6 M parameters, 66.2 GFLOPs and 0.07 s per image on the authors' hardware 2111.11843 tab. 3. Card: U-shape Transformer.

Paired CNN with RGB and HSV branches for underwater colour

Three CNN blocks trained end to end on paired underwater photos, with no adversarial term. An RGB block removes noise and colour cast. An HSV block predicts four piecewise-linear curves, for value, saturation and hue, from global pooling 2103.07138 §III-A eq. 1. An attention block takes the input and both outputs and predicts six weight maps, which blend the RGB and HSV outputs per pixel and per channel 2103.07138 §III-A. The loss sums L1, SSIM, an HSV term and a VGG-19 perceptual term 2103.07138 §III-B eq. 2. Training uses 800 real pairs from UIEBD and 1200 synthetic pairs built from NYU-v2 RGB-D images 2103.07138 §IV-A. The paper says the model has far fewer parameters than deep CNNs but gives no count, runtime or memory figure 2103.07138 §I. Card: UIEC^2-Net.

Author-reported results on 90 real and 900 synthetic test images: PSNR 24.5663 and SSIM 0.9346 on the real set, and PSNR 20.5442 and SSIM 0.8486 on the synthetic set, the best full-reference scores in its tables [arxiv:2103.07138 tab. II] tab. I. On real images it also reports the best UCIQE, UISM and UIQM among the compared methods 2103.07138 tab. III. Its own ablation shows the RGB block alone reaches PSNR 23.8228 on the real set, against 24.5663 for the full model 2103.07138 tab. IV.

When to use which

Compare numbers only on the same test set. The four variants report on different datasets and metrics, so their scores do not rank against each other.

Implementation sketch

The common core is one forward pass: predict a correction from the image, apply it, and clamp to the valid range. The variants differ in how the correction is built and applied.

# Common core of image enhancement  [arxiv:2205.14871 §3.2] [arxiv:2303.17569 §3.1]
# Input:  sRGB image in [0, 1], shape (B, 3, H, W)
# Output: enhanced sRGB image in [0, 1]

def enhance(image):
    # Variant A: predict a per-pixel multiply and add, and a global colour
    # matrix and gamma  [arxiv:2205.14871 §3.2] [eq. 4]
    multiply_map, add_map = local_branch(image)
    colour_matrix, gamma = global_branch(image)
    adjusted = image * multiply_map + add_map
    coloured = apply_colour(colour_matrix, adjusted)
    out = clamp(coloured, 1e-8, 1.0) ** gamma
    # Variant B: predict one illumination map and divide by it  [arxiv:2303.17569 §3.1]
    # illumination = enhancer(image)           # (B, 1, H, W)
    # out = image / (illumination + epsilon)   # keep a small epsilon guard
    # Variant C: U-shaped generator, no division or gamma  [arxiv:2111.11843 §3.2.1]
    # out = generator(image)                   # trained with GAN and colour losses
    # Variant D: RGB block, HSV curves, then per-pixel blend  [arxiv:2103.07138 §III-A]
    # rgb_out = rgb_block(image)               # (B, 3, H, W)
    # hsv_out = hsv_curves(rgb_out)            # (B, 3, H, W), curves from pooling
    # w = attention_block(concat([image, rgb_out, hsv_out], dim=1))  # (B, 6, H, W)
    # out = w[:, :3] * rgb_out + w[:, 3:] * hsv_out
    return clamp(out, 0.0, 1.0)

Notes on the variants:

• Variant A's gamma needs a floor on the base before the power, otherwise the power is undefined at zero 2205.14871 §3.2 eq. 4.
• Variant B's division is sensitive near zero, so keep an epsilon in the denominator and clamp the result 2303.17569 §3.1.
• Variant B's CLIP scoring depends on a logit scale that the paper's equations do not show. Without it, the margin used in refinement cannot be reached 2303.17569 §3.2 eq. 2 eq. 6. Keep the scale when you implement.
• Variant B's enhancer is not a standard encoder-decoder U-Net, despite the paper's wording. Do not build one from that phrase alone 2303.17569 §3.1.
• Variant C's loss weights in eq. 13 are tied to the reduction the released code uses. The colour cross-entropy sums over pixels there. With a per-pixel mean, the same weights would count that term far less 2111.11843 §3.3 eq. 13 LAB.py.
• Variant C's released generator has no output activation, so the output is unbounded. Clamping it, or adding a sigmoid, changes what is learned block.py.
• Variant D's curve sum in eq. 1 needs a knot beyond the last one, which the paper does not define. The released code sums one step fewer, with M = 11 knots 2103.07138 §III-A eq. 1 UIEC2Net.py.
• Variant D's attention prose lists the HSV output first, but its channel description, and the released code, give the first three maps to the RGB output 2103.07138 §III-A UIEC2Net.py.
• Variant D's RGB to HSV conversion divides by V and by V − min. The paper gives no grey-case handling, so keep a small epsilon in both denominators 2103.07138 §III-C eq. 9 eq. 10 RGB2HSV.py.

Contradictions

• CLIP-LIT's PSNR claim. Its main table says the method has the best PSNR on the BAID test set, at 21.579 2303.17569 tab. 1. Its retrained supervised table gives Zhao et al. a BAID PSNR of 22.123, above that value 2303.17569 tab. 7. The same paper gives both numbers, and this page does not resolve which is the right comparison.
• IAT's efficiency and quality claim. The text says the method is best on both quality and efficiency on LOL 2205.14871 §4.1. The same paper's table gives MAXIM higher PSNR and SSIM on LOL-V1 (23.43 and 0.863, against 23.38 and 0.809) 2205.14871 tab. 1.
• IAT's exposure average. The table gives an average PSNR of 20.34 for IAT on the exposure set 2205.14871 tab. 3. The five per-expert values in the same paper average to 20.40 2205.14871 tab. 3. This page quotes the table value.
• The underwater paper's "more than 2 dB" claim. The abstract says the method is more than 2 dB better 2111.11843 abstract. From its table, the gap is 2.13 dB on Test-U90 against Ucolor, but 1.25 dB on Test-L400 2111.11843 tab. 3. The claim holds only on the first set.
• The underwater paper's efficiency claim. The results text says the model has relatively few parameters, FLOPs and running time 2111.11843 §4.5. Its own table gives 66.2 GFLOPs and 65.6 M parameters, above FUnIE and UIE-DAL. Its runtime is below WaterNet, Ucolor and FUnIE, but UGAN is faster 2111.11843 tab. 3.
• UIEC^2-Net's training loss, paper against code. The paper trains with L1, SSIM, an HSV term and a perceptual term 2103.07138 §III-B eq. 2. The shipped training script computes only the SSIM loss on the network output train.py. The released config also uses other weights and a 1000-epoch schedule, against 50 epochs in the paper 2103.07138 §IV-A. The page follows the paper, and the code is a different run.
• The underwater dataset size. The LaTeX source gives 4279 image pairs 2111.11843 §3.1. The arXiv abstract metadata gives 5004. This page uses 4279.
• Across the four papers. Their datasets, metrics and training data differ, so no result contradicts another. Do not read one as a ranking of the variants.

Papers

• IAT: per-pixel and global adjustment with about 0.09M parameters, trained on paired data.
• CLIP-LIT: unpaired training with learned CLIP prompts and an illumination map, evaluated on backlit photos.
• U-shape Transformer: paired transformer GAN for underwater colour, with the LSUI dataset.
• UIEC^2-Net: paired CNN with RGB and HSV branches and per-pixel attention, for underwater colour.