Iterative Prompt Learning for Unsupervised Backlit Image Enhancement

In plain words

Photos taken against a strong light source come out with very uneven illumination, so some areas are over-brightened and others stay too dark. Fixing this by hand is slow. Automatic fixers that learn from pairs of bad and good photos need data that is hard to get, and the simpler ones make assumptions about brightness that do not hold in every scene.

This work uses CLIP, a model trained to match images with text, as a judge of how a photo looks. Plain words such as "backlit" or "normal light" are poor judges, because the right word changes from photo to photo. So the method learns the pair of text prompts itself, one for "bad light" and one for "good light". It then trains a small network that brightens a photo by dividing it by a per-pixel light map, scored against those learned prompts. The prompts and the network take turns being improved. The training data are two unpaired sets of photos, one backlit and one well-lit, with no matching pairs. The authors report better scores than earlier methods on their test sets.

TL;DR

CLIP-LIT enhances backlit photos without paired training data. It learns a positive and a negative text prompt in CLIP space, trains an enhancement network that outputs an illumination map, and alternates prompt refinement with network tuning. Trained on 380 backlit and 384 well-lit images, it reports higher PSNR and MUSIQ than the listed baselines on the BAID test set (author-reported). The released code departs from the paper in loss weights, margins, padding and optimizer settings, so read Paper vs code before implementing.

Problem

Backlit images have highly imbalanced illumination, which hurts visual quality and downstream perception §1. Supervised enhancers trained on paired data tend to brighten well-lit regions too much or under-correct dark ones §1 fig. 3. The paired backlit set BAID, which the paper uses for evaluation, has ground truth retouched by photographers, and the paper says those targets are themselves sub-optimal §1 §2. Earlier unsupervised methods rely on curves built from physical assumptions, or on adversarial training against well-lit images, and the paper judges their robustness and generalisation as limited §1 §2. CLIP can partly tell backlit from well-lit images, but a hand-picked word prompt is unreliable: the best wording varies from image to image, and labelling it per image is costly §1 fig. 2. The method is meant for settings with only unpaired images and a frozen CLIP model §3.

Method

Overview §3. Training has two stages. Stage 1 learns an initial positive and negative prompt pair and trains the enhancer against it. Stage 2 alternates prompt refinement, which uses a ranking loss, with tuning of the enhancer. CLIP stays frozen throughout. The paper says the loop repeats until the results look visually pleasing, and it gives no numeric stopping rule §3 §3.2.

Prompt initialisation §3.1. Each prompt is a block of N = 16 learnable token embeddings of width 512, passed through CLIP's text encoder §3.1 §4 Training. Backlit and well-lit training images pass through CLIP's image encoder. The cosine similarity between an image and the two prompts goes through a softmax, and the result is the probability of the positive prompt eq. 2. The pair is fitted with a binary cross-entropy: label 0 for backlit, 1 for well-lit eq. 1 §3.1.

Enhancer §3.1. The enhancer predicts a one-channel illumination map I_i. The output is I_t = I_b / I_i, a division inspired by Retinex §3.1. The paper calls the enhancer a simple U-Net and says more complex networks would also work §3.1. See Paper vs code for what the released code builds.

Enhancer losses §3.1. The CLIP-enhance loss L_clip is the softmax probability that the enhanced result matches the negative prompt eq. 3. The loss is minimised, so the result is pushed away from the negative prompt. The identity loss L_identity compares features of the backlit input and the output at five layers of a ResNet-101 CLIP image encoder eq. 4. The total is L_enhance = L_clip + 0.9 · L_identity eq. 5. Training has two parts: first identity loss only, to reconstruct the input, then both losses §3.1. The layer weights are 1.0 for every layer during self-reconstruction. During enhancer training they are 1.0 for layers 0 to 3 and 0.5 for layer 4, which the authors chose because the last layer relates most to colour §3.1.

Prompt refinement §3.2. The negative score S(I) is the probability of the negative prompt, as in eq. 6. Refinement uses a margin ranking loss eq. 7 with three hinge terms: well-lit below backlit by margin m0, the enhanced output below backlit by m0, and the enhanced output close to well-lit by margin m1. So that each round improves on the last, the previous round's output I_{t-1} also stays in the loss eq. 8. This adds a term that asks the current output to score at least m2 below the previous output, with m2 = m1 §3.2. The margins are m0 = 0.9 and m1 = 0.2, chosen empirically §3.2. The appendix says the theoretical value m1 = 0 did not converge in their runs app. B.1.

Enhancer tuning §3.2. The enhancer is trained as in stage 1, but with the refined prompts. Its new outputs become the data used in the next refinement round §3.2.

Scheduling app. B.2. Stage lengths are controlled by thresholds rather than fixed counts alone. Prompt learning stops once its loss falls below Thr_A = 60, and enhancer training stops once its loss falls below Thr_B = 90. If neither threshold is reached, the stage switches after 1000 iterations [app. B.2].

# Iterative prompt learning for unsupervised backlit enhancement  [§3]
# Input:  backlit images, well-lit images (unpaired), frozen CLIP
# Output: enhancer network, prompt pair (negative, positive)
# hinge(x) = max(0, x)

def iterative_prompt_learning(backlit, well_lit):
    # Identity-only warm-up of the enhancer, 1K iterations  [§3.1] [§4]
    enhancer = self_reconstruct(backlit, iters=1000)
    # Initial prompt pair from cross-entropy on CLIP similarity  [§3.1] [eq. 1]
    prompt_neg, prompt_pos = learn_initial_prompts(backlit, well_lit)
    # Enhancer trained against the frozen prompts  [§3.1] [eq. 5]
    enhancer = train_enhancer(enhancer, backlit, prompt_neg, prompt_pos)
    # Repeat until visually pleasing; no numeric stopping rule is given  [§3]
    while not visually_pleasing(enhancer):
        # Refine the prompts with the ranking loss  [§3.2] [eq. 7] [eq. 8]
        prompt_neg, prompt_pos = refine_prompts(
            backlit,
            well_lit,
            enhancer,
        )
        # Tune the enhancer against the refined prompts  [§3.2]
        enhancer = train_enhancer(enhancer, backlit, prompt_neg, prompt_pos)
    return enhancer, prompt_neg, prompt_pos

# Enhancement: divide the input by a one-channel illumination map  [§3.1]
def enhance(enhancer, backlit):
    # Illumination map, shape (B, 1, H, W), predicted by the enhancer  [§3.1]
    illumination = enhancer(backlit)
    # Enhanced result I_t = I_b / I_i  [§3.1]
    return backlit / illumination

# Negative score S: softmax over cosine similarity to the two prompts  [§3.2]
# No temperature is stated in the paper  [eq. 2] [eq. 6]
def negative_score(image_feature, prompt_neg, prompt_pos):
    # Cosine similarity of the image to each prompt, then softmax over the pair
    # Returns the probability of the negative prompt
    return softmax_over_prompts(image_feature, prompt_neg, prompt_pos)[0]

# Enhancer objective  [§3.1] [eq. 3] [eq. 4] [eq. 5]
def enhancer_loss(enhanced, backlit, prompt_neg, prompt_pos):
    # Negative-prompt probability of the enhanced result, minimised  [eq. 3]
    clip_term = negative_score(enhanced, prompt_neg, prompt_pos)
    # Feature distance to the backlit input at 5 layers of RN-101 CLIP,
    # layer weights 1.0 (layers 0-3) and 0.5 (layer 4)  [§3.1] [eq. 4]
    identity_term = identity_distance(enhanced, backlit)
    # Balance weight w = 0.9  [§3.1] [eq. 5]
    return clip_term + 0.9 * identity_term

# Prompt refinement loss on negative scores  [§3.2] [eq. 7] [eq. 8]
def refinement_loss(score_backlit, score_well, score_enh, score_prev):
    # Well-lit should score below backlit by m0 = 0.9  [§3.2]
    loss = hinge(score_well - score_backlit + m0)
    # Previous-round output should also score below backlit by m0  [eq. 8]
    loss = loss + hinge(score_prev - score_backlit + m0)
    # Enhanced output should stay close to well-lit, margin m1 = 0.2  [§3.2]
    loss = loss + hinge(score_well - score_enh + m1)
    # Enhanced output should score m2 below the previous output  [eq. 8]
    loss = loss + hinge(score_enh - score_prev + m2)
    return loss

Hyperparameters

Values are as stated in the paper unless the Where column names the code. Table and figure numbers were counted from the LaTeX source, not from a compiled PDF.

Results

Author-reported. Setup: BAID test set (368 backlit images, retouched references), Backlit300 test set (305 images, scored with MUSIQ only), and five extra unseen sets §4 app. F. Training uses 380 BAID backlit and 384 DIV2K well-lit images, unpaired §4 Dataset. All unsupervised methods in Table 1 were retrained on the same unpaired data §4 Compared Methods. Baselines are the authors' released models unless the row says "retrained" [§4 Compared Methods].

Table 1. BAID test set tab. 1

Headline: CLIP-LIT has the best PSNR, LPIPS and MUSIQ in the table. SSIM ties Zero-DCE++ at 0.883 tab. 1 §4.1.

Table 2. Backlit300 test set, MUSIQ only tab. 2

Five extra unseen sets, MUSIQ tab. 8 app. F

Ablation on iterative learning, BAID test tab. 3 §4.3

Training reference data, BAID test tab. 4 §4.3

CLIP-enhance loss vs adversarial loss, BAID test tab. 5 §4.3

Prompt initialisation, BAID test tab. 6 app. A.2

Retrained supervised methods tab. 7 app. D

User study §4.2 fig. 11. 40 participants chose the preferred output for 20 Backlit300 images. Each comparison pitted CLIP-LIT against one of Zero-DCE, ExCNet, URetinex-Net or EnlightenGAN. The authors report that over 60% of votes went to CLIP-LIT on each image.

Qualitative results. The paper's figures compare outputs on BAID and Backlit300 fig. 8 fig. 9 and show the supplementary sets app. E app. F. Figures are described here, not reproduced: see the paper.

Speed app. H. The authors report 4K images in about 0.5 s on one RTX 3090.

Paper vs code

Code: official repository ZhexinLiang/CLIP-LIT at commit 25ed5be6 (linked from the project page, owned by the first author). The repository has no LICENSE file. Its README mentions an S-Lab License in a commented-out section, so the code's terms are unclear and nothing from it is quoted here. Its training and test scripts were read to compare them with the paper.

• Identity distance. The paper uses an L2 norm of the feature difference eq. 4. The code uses a squared-error mean per layer, summed over the batch clip_score.py. The L2 helper exists but is unused clip_score.py.
• Loss weights. The paper gives L_clip + 0.9 · L_identity eq. 5. The code uses 320 · L_clip + 0.9 · 25 · L_identity train.py [code: train.py#L236] train.py. The identity term is about 13 times weaker relative to CLIP than the paper's ratio. The CLIP term is averaged over the batch, but the identity term is summed, so the balance also depends on batch size clip_score.py clip_score.py.
• Reconstruction. Self-reconstruction uses a scale of 25 on the identity term with all five weights set to 1.0 train.py. The paper gives no scale for this stage.
• Logit scale. The similarity inside the softmax is multiplied by 100 clip_score.py train.py train.py. The paper's equations have no temperature eq. 2 eq. 6.
• Margins. The paper fixes m0 = 0.9 and m1 = 0.2 §3.2. The code uses (1.0, 0.2) in the first refinement round, (0.9, 0.2) after one stored output, and (0.9, 0.1) once two outputs are stored train.py [code: train.py#L361] train.py.
• Refinement loss scale. The code multiplies the refinement loss by 200 and averages it over the batch train.py clip_score.py. The paper does not state a scale or a reduction.
• Prompt-initialisation loss. The code uses two-class cross-entropy on the scaled logits train.py train.py. The paper writes a binary cross-entropy on the softmax probability eq. 1 eq. 2.
• Labels. Well-lit versus backlit is decided by file path: names containing "normal", "output", "/data/HQ/", "DIV2K" or "high" count as well-lit dataloader_prompt_add.py dataloader_prompt_add.py. The paper does not describe this rule.
• Optimizer. The paper states β2 = 0.99 §4 Training. The code uses the default betas (0.9, 0.999) train.py train.py.
• Batch sizes. The paper's sentence order suggests prompt 8 and enhancer 16. The code uses enhancer 8 and prompt 16 train.py [code: train.py#L441].
• Iteration counts. The paper gives 1K reconstruction and 10K prompt initialisation iterations §4 Training. The code's from-scratch fallback is 8K for prompt initialisation train.py train.py train.py. The enhancer stage defaults to 2100 iterations, then 1000 prompt iterations, both extended by thresholds train.py train.py train.py train.py. The paper says 1000 per stage app. B.2.
• Enhancer architecture. The paper says "simple U-Net". The code's network is a stack of residual blocks with a pyramid pooling module, at full resolution. It has no strided down-sampling and no skip connections in its forward pass model_small.py model_small.py.
• Padding at train versus test. Training builds the variant with zero padding train.py. The test script builds the variant with reflect padding and loads the same released weights test.py model_small.py. The paper does not say which variant produced the released weights.
• Test-time resizing. The paper resizes test images whose sides exceed 2048 px §4 Inference. The released test script does not resize; it runs at native resolution test.py.
• Output clamp and epsilon. Training divides by (map + 1e-9) and clamps the result to [0, 1] train.py. The test script clamps but adds no epsilon test.py. The paper states neither.
• Data resolution. The enhancer trains on 512 px images [code: dataloader_images.py#L131]. The prompt and refinement loaders use 256 px dataloader_prompt_add.py dataloader_prompt_margin.py, then resize to 224 for CLIP clip_score.py. The paper gives only 512.
• Prompt initialisation from scratch. The paper says the prompts are random §3.1. The default from-scratch path starts both prompts from the same token embedding of "X" repeated N times train.py. The paper's random and word-based settings are compared in app. A.2.
• Pretrained files. The default training configuration loads an initial prompt pair, init_prompt_pair.pth. The clone does not include it. The README says it is a separate release download train.py [README.md].
• Unused settings. grad_clip_norm, T_max and eta_min are parsed but never used train.py train.py train.py. random_crop is defined but never called train.py.
• Undocumented behaviour. The code re-initialises the enhancer if a running PSNR-based score stays low after 100 iterations, and then adds 100 to the reconstruction count train.py train.py [code: train.py#L281]. It also keeps the checkpoint with the best running score train.py train.py. The paper mentions neither.
• Pretrained checkpoint. The released enhancer enhancement_model.pth is included in the repository enhancement_model.pth.

Pitfalls

• The paper's equations cannot reach its margin. Without a temperature, the negative score lies between about 0.12 and 0.88 when the cosine similarities are in [−1, 1]. The difference between well-lit and backlit scores is then at most about 0.76, below m0 = 0.9, so the first hinge never reaches zero eq. 2 eq. 6 §3.2. The code's scale of 100 is what makes m0 = 0.9 reachable clip_score.py. Keep the scale when you implement.
• The identity balance is a choice the paper does not show. If you follow eq. 5 literally, the identity term carries about 13 times more weight than in the code. The code's outputs are tuned to its own scaling.
• The m2 hinge does not do what the text says. The text says the new outputs are kept "similar" to the previous ones §3.2. Eq. 8 instead requires the current output's negative score to be at least m2 below the previous one, which pushes it toward well-lit, not toward the previous output eq. 8. The code has the same one-sided form train.py [code: clip_score.py#L162].
• Eq. 3 is minimised, though the text calls it a similarity. The loss pushes the enhanced result away from the negative prompt eq. 3 §3.1 [code: train.py#L235].
• m1 = 0 is the theoretical value. The appendix says the loss did not converge at 0, so 0.2 was used app. B.1. The code later drops it to 0.1 train.py.
• Thresholds are in different loss units. Thr_A = 60 and Thr_B = 90 are compared against the code's scaled losses, not the paper's app. B.2 [code: train.py#L292] train.py. The prompt-initialisation check uses the unscaled cross-entropy, while refinement uses the ×200 loss [code: train.py#L321] train.py. Thresholds from the paper cannot be transferred to a different loss scale.
• No stopping rule. The paper stops "when visually pleasing" §3. The code runs for a fixed number of epochs with threshold extensions train.py. The paper's 50K total is not reproduced by a stated setting §4 Training.
• The Retinex divide is numerically sensitive. The enhancer's output is non-negative because of its final ReLU, so the map can reach zero. Training guards this with 1e-9, but the test script does not train.py test.py model_small.py. Add the same guard to inference.
• Padding mismatch. Weights trained with zero padding are run with reflect padding at test time train.py test.py. Expect small border differences. Pick one variant and use it for both training and evaluation.
• Resolution changes the numbers. The paper resizes large test images to a 2048 px long side for all methods §4 Inference. The released code does not. Metrics on very large images may differ from the paper's.
• MUSIQ is the main metric for backlit data. The authors say the full-reference metrics do not fully judge backlit enhancement, so they lean on MUSIQ §4 Quantitative Comparison. BAID references are retouched by photographers and sub-optimal §2.
• Batch sizes are ambiguous in the paper. The text gives "8 and 16" without saying which goes with which stage §4 Training. The code uses enhancer 8 and prompt 16 train.py train.py.
• Adam betas differ. Implement β2 = 0.99 if you follow the paper. The code runs with 0.999 §4 Training train.py.
• The enhancer is not a standard U-Net. Do not build an encoder-decoder from the word "U-Net". The released network keeps full resolution and has no skip connections model_small.py.
• Costs the abstract hides. Each enhancer update passes the enhanced output through the ViT-B/32 image encoder at 224 px for the CLIP term, and through the RN-101 CLIP twice (output and input) for the identity term [code: clip_score.py#L74-L87] clip_score.py [code: clip_score.py#L102]. The paper says the frozen CLIP adds no burden beyond prompt initialisation and refinement §3. Its cost per iteration is not stated.
• Training data are small. Training uses 380 backlit and 384 well-lit images. The paper varies only the well-lit reference set tab. 4 §4.3, and no experiment varies the backlit set.
• Code licence. No LICENSE file is present and the README's licence section is commented out. Treat the code as unlicensed for reuse. Implement from the paper and cite code by path only.

Lineage

• Builds on: CLIP for image-text similarity and its ResNet-101 layers §3.1. Prompt learning with CoOp and CoCoOp for vision-language adaptation §2. Retinex model for the division-based output §3.1. The paper's "U-Net" for the enhancer §3.1. No card exists here for these, so no link is given.
• Compared against: the released and retrained methods in tab. 1 and tab. 2: Afifi et al., Zhao et al., URetinex-Net, SNR-Aware, Zero-DCE, Zero-DCE++, RUAS, SCI, EnlightenGAN, ExCNet.