Zero-Reference Low-Light Enhancement via Physical Quadruple Priors

In plain words

Dark photos are hard to fix with a model that has only seen dark photos, because the model then learns the particular lighting of its training set. This paper trains without any dark photos. It uses only ordinary well-lit pictures.

The trick is to describe a picture by something that does not depend on how much light was on the scene: roughly, the colour and the local shape of its brightness, with the overall light level left out. The model learns to rebuild a normal-looking picture from that description. At test time, a dark photo is described the same way, and the model rebuilds it with ordinary lighting. A large pretrained image generator does most of the rebuilding, and a second decoder keeps fine detail such as text sharp.

A lighter version of the system learns to do the same job in a single pass, so it can run much faster.

TL;DR

• Zero-reference: trained only on normal-light COCO images, no low-light pairs or unpaired low-light sets §1 §3.1.
• An illumination-invariant "physical quadruple prior" (H, C, W, O) is the only input the generator sees §2.2.
• Frozen Stable Diffusion 1.5 with a ControlNet-style prior branch and a bypass decoder §2.3.
• Author-reported: best unsupervised LOL PSNR and best unpaired BRISQUE in tab. 1; the code is not licensed, so cite only.

Problem

Zero-reference enhancers such as Zero-DCE, RUAS and SCI need hand-tuned parameters and change their output with the training set; SCI over- or under-exposes on different data §1 fig. 1. Supervised Retinex-style and diffusion models need paired data and overfit to their training domain §3.2. The paper argues that a model must know what "normal illumination" looks like without seeing low-light data §1.

Method

Overview. Training uses normal-light images only. The network learns to rebuild an image from its illumination-invariant prior. At test time, a low-light image yields a prior, and the same network rebuilds it as a normal-light image §2.1 fig. 2.

Physical quadruple prior §2.2 fig. 3. The prior has four parts, derived from the Kubelka–Munk light-transfer model eq. 1. Per-pixel spectral quantities E, E_λ and E_λλ are estimated from RGB by a learned 3×3 linear map, followed by Gaussian colour-derivative filters §2.2.

1. H = arctan(E_λ / E_λλ). Derived assuming equal-energy illumination §2.2.
2. C = log((E_λ² + E_λλ²) / E²). Derived assuming a matte surface, so specular reflection vanishes §2.2.
3. W = tan(|∂E/∂x · 1/E|). Derived assuming uniform illumination §2.2. The spatial derivative is taken in x and y, with magnitude √(E_x² + E_y²) §2.2.
4. O = the order of the R, G and B channels, one value per channel, normalised to [-1, 1] §2.2.

H, C, W and O are concatenated, giving 1 + 1 + 1 + 3 = 6 channels §2.2. In the code the 3×3 map is a learnable parameter with fixed starting values §2.2 ciconv2d.py. The filter scale σ is predicted from the input image §2.2.

Prior-to-image generator §2.3 fig. 3. Stable Diffusion v1.5 is frozen. A ControlNet-style branch takes the prior as its condition §2.3. The text prompt is always the empty string, so no caption is needed §2.3. Training:

1. Encode the clean image I with the frozen SD encoder to a latent z0 §2.3.
2. Sample a timestep t and noise ε, and form z_t from z0 with the standard forward-noise formula §2.3.
3. Predict ε from z_t and the prior, and minimise a loss on ε and on z0 §2.3.

Bypass decoder §2.3 fig. 4 fig. 5. The SD decoder distorts detail such as text and faces. The paper adds encoder skip features from a randomly distorted copy of the image (illumination jitter and noise) to the decoder, with zero- or identity-initialised fusion layers §2.3. At test time, skip features come from the input image itself, so the output keeps the input's detail §2.3 fig. 6.

Denoising §2.3. During training only, Gaussian-Poisson compound noise is applied to the input before the prior is computed §2.3.

Distillation §2.3. A lightweight CNN with residual blocks and transformer blocks at the bottleneck is trained with L1 loss on 1.7k samples generated by the full framework §2.3. It replaces the diffusion sampling loop with a single forward pass §1 §3.3 fig. 11. The paper says more details are in the supplementary.

Training step and test-time inference, in pseudocode. Names are ours.

# Physical quadruple prior  [§2.2] [code: ciconv2d.py#L86-L156]
# Input: rgb (B, 3, H, W) in [0, 1]; output: prior (B, 6, H, W)

def quadruple_prior(rgb):
    # Scale sigma per image from a small CNN, clamped to +-2.5
    # [code: ciconv2d.py#L96-L102] [code: ciconv2d.py#L123-L124]
    sigma = clamp(mean(small_cnn(rgb)), -2.5, 2.5)
    # Linear 3x3 map from RGB to E, E_lambda, E_lambda_lambda  [§2.2]
    # [code: ciconv2d.py#L93] [code: ciconv2d.py#L129]
    spectral = colour_matrix @ rgb
    # Gaussian filters at scale 2^sigma; derivative kernels are
    # normalised by their absolute sum  [code: ciconv2d.py#L15-L35]
    E, E_x, E_y = gaussian_derivative(spectral[0], sigma)
    E_l, _, _ = gaussian_derivative(spectral[1], sigma)
    E_ll, _, _ = gaussian_derivative(spectral[2], sigma)
    # H = arctan(E_lambda / E_lambda_lambda)  [§2.2]
    H = atan(E_l / (E_ll + eps))
    # C = log((E_lambda^2 + E_lambda_lambda^2) / E^2)  [§2.2]
    C = log((E_l**2 + E_ll**2) / (E**2 + eps) + eps)
    # W (code form): atan of squared normalised spatial derivatives
    # [code: ciconv2d.py#L67-L70] [code: ciconv2d.py#L153]
    W = atan((E_x / (E + eps))**2 + (E_y / (E + eps))**2)
    # O: +1 for the largest channel, -1 for the smallest, 0 otherwise
    # [code: ciconv2d.py#L108-L121]
    O = channel_order(rgb)
    return concat([H, C, W, O], dim=1)   # 1 + 1 + 1 + 3 = 6 channels


# One training step  [§2.3] [code: cldm/cldm.py#L293-L368]
# Input: normal-light image I in [0, 1], cropped to 512
# [code: coco_dataset.py#L45-L48]

def training_step(I):
    # Latent of the clean image from the frozen SD encoder  [§2.3]
    z0 = frozen_vae_encode(2 * I - 1)            # (B, 4, 64, 64)
    # Prior of a degraded copy; noise applied with p = 0.5
    # [code: coco_dataset.py#L58]
    prior = quadruple_prior(maybe_add_noise(I))  # (B, 6, 512, 512)
    # Random timestep, noise, and noisy latent  [§2.3]
    t = sample_timestep()
    noise = sample_gaussian_like(z0)
    z_t = sqrt(alpha_bar[t]) * z0 + sqrt(1 - alpha_bar[t]) * noise
    # Frozen U-Net with prior features added through zero convolutions
    noise_hat = unet_with_prior(z_t, t, prior)
    # Released code: noise term only; paper also adds an L_z0 term
    # [code: ldm/models/diffusion/ddpm.py#L387-L412]
    return mse(noise_hat, noise)


# Inference  [code: test.py#L70-L110]
# Input: low-light image; output: enhanced image

def enhance(low_light):
    # Resize: short side 512, sides multiple of 64  [code: test.py#L72]
    image = resize_short_side(low_light, 512)
    # Prior from the clean input in [0, 1]  [code: test.py#L76-L78]
    prior = quadruple_prior(image / 255)
    # Skip features from the input for the bypass decoder [code: test.py#L81]
    skips = bypass_encoder_features(image * 2 - 1)
    # Sample the latent from noise, 10 DPM-Solver steps
    # [code: test.py#L98-L102]
    z0_hat = dpm_solver(prior, steps=10, order=3, guidance=9.0)
    # Decode with the bypass decoder  [§2.3] [code: test.py#L107-L110]
    return bypass_decode(z0_hat, skips)

Hyperparameters

Results

Author-reported. Setup: LOL v1 and v2 test sets (115 pairs), MIT-Adobe FiveK (500 test pairs, split as in Retinexformer), and five unpaired sets (LIME, NPE, MEF, DICM, VV) §3.1. The method is trained on COCO only. Baselines are supervised LOL- or MIT-trained models and nine unsupervised methods §3.1 tab. 1.

Supervised reference points: Retinexformer trained on LOL reaches 28.48 PSNR on LOL but 13.87 on FiveK tab. 1. The paper's claim that the method beats most unsupervised methods and narrows the gap to supervised ones is supported by tab. 1 §3.2.

Ablations on LOL tab. 2: removing H, C, W or O lowers PSNR to 17.60, 17.60, 17.77 and 18.63 tab. 2 §3.3. The paper's own "final version" row gives 20.25 PSNR and 0.807 SSIM, which differs slightly from tab. 1 (see Paper vs code). Replacing the SD decoder with the bypass decoder improves LOL PSNR from 19.26 to 20.25 tab. 2.

Lightweight model, LOL/MIT: PSNR 20.45/18.15, SSIM 0.798/0.770, LPIPS 0.290/0.175, LOE 0.273/0.174 §3.3. The paper reports speed as 500× faster than the full model and 0.03 s for a 1024×1024 image on a Tesla M40; it does not give the baseline for the speed-up §3.3.

Paper vs code

Official repository: https://github.com/daooshee/QuadPrior README.md, pinned at code_commit above. The paper's project page (abstract link) links to this repository, and the README links the first author's homepage (daooshee.github.io/website), so the owner is an author README.md. No LICENSE file is present, so the code is cite-only: nothing is copied from it. The arXiv metadata has no comments field to check.

• Training loss. Paper: L_DIFF = L_z0 + L_noise §2.3. Code: the training loss is only the ε mean-squared error, with l_simple_weight=1 and original_elbo_weight=0 by default ddpm.py ddpm.py. The config does not override these cldm_v15.yaml. No z0 term was found in the training path cldm.py.
• W invariant. Paper: W = tan(|∂E/∂x · 1/E|) §2.2. Code: arctan of the sum of squared normalised x and y derivatives, with no square root and no tangent ciconv2d.py ciconv2d.py.
• Order channels O. Paper: normalised to [-1, 1] §2.2. Code: the largest channel gets +0.5 twice and the smallest −0.5 twice, so values are exactly −1, 0 or +1 ciconv2d.py. The two scatter calls hit the same index, so the duplicate adds nothing new.
• Derivative filters. Paper: Gaussian colour derivative filters §2.2. Code: the x and y derivative kernels are normalised by the sum of their absolute values, not the Gaussian derivative scale ciconv2d.py. This changes the magnitude of Ex and Ey, and so of W.
• Scale σ. Paper: predicted from the input, architecture not stated §2.2. Code: 3→16→16→1 convolutions with SiLU, averaged over channels and space per image, clamped to ±2.5 ciconv2d.py ciconv2d.py. The filter std is 2^σ ciconv2d.py.
• Prior input channels. Code: the first hint convolution takes 6 channels, matching H, C, W and the three O values cldm.py.
• Noise. Paper: noise is applied while extracting the prior during training §2.3. Code: noise augments only the hint copy during training, with p=0.5 coco_dataset.py coco_dataset.py. Test-time input is clean test.py.
• Text prompt. Paper: always empty §2.3. Code: a precomputed CLIP embedding of the empty string is used for every sample cldm.py cldm.py compute_empty_prompt.py.
• Starting point of training. Paper: not stated. Code: resume_path defaults to a released checkpoint, so training continues from it rather than from SD and ControlNet initialisation train.py train.py. Starting from scratch means editing this line.
• Training length. Paper: 140k steps, about 5 epochs §3.1. Code: no step limit; a checkpoint is saved every 10k steps train.py. The run that produced the 140k figure is not identifiable from the code.
• Sampler. Paper: "DPM-Solver++ ... 10 steps" appears only in the distillation paragraph §2.3. Code: DPMSolverSampler multistep with order 3 and 10 steps test.py sampler.py.
• Guidance. Paper: no guidance scale stated. Code: scale 9.0 test.py test.py. See Pitfalls: in this code the guidance has no effect.
• Bypass decoder. Code: the fusion layers use an identity-like init for the 3×3 merge and a zero-output init for the residual block models.py, consistent with "zero or self" §2.3. The decoder checkpoint used at test time is main-epoch=00-step=7000.ckpt test.py. Its training loop, data and distortion are not in the repository: my_vae/autoencoder.py has only a forward pass and a demo block (from line 69) that imports a data module not in the repository autoencoder.py.
• Lightweight model. Code: no distilled network, distillation script or L1 training config is in the repository. The 1.7k-sample dataset is not released. Its architecture and training are therefore unverified.
• Results table vs ablation. Table 1 gives LOL 20.31 PSNR, 0.808 SSIM, 0.202 LPIPS, 0.281 LOE for this paper tab. 1. The ablation's "final version" gives 20.25, 0.807, 0.199 and 0.278 tab. 2. Use tab. 1 as the headline and note the mismatch.
• Supplement. Implementation detail, showcases, colourisation results and lightweight scores are referred to the supplementary §3.1 §3.2 §3.3, which is not in the LaTeX source. Those values are unverified here.

Pitfalls

• Guidance is a no-op in the released code. Both branches of the classifier-free formula go through the same apply_model, which always uses the empty-text embedding and the same control image, so the conditional and unconditional predictions are identical cldm.py dpm_solver.py. The scale 9.0 in test.py therefore changes nothing as written. Verify before relying on it. A re-implementation that uses true guidance will change the output.
• Which loss. The paper's z0 term is absent from the code. Matching the paper means adding the L_z0 term; matching the code means dropping it. The paper claims it speeds convergence §2.3; this is not tested here.
• Equation numbers. Only eq. 1 (the Kubelka–Munk model) is numbered with confidence in this card. The invariant and loss formulas are cited by section, because the numbering of the unlabelled display lines was not verified against the compiled PDF.
• Per-image kernel size. σ is predicted per image, so the Gaussian kernel radius, ceil(3·2^σ + 0.5), differs from image to image ciconv2d.py ciconv2d.py. Batched re-implementations must handle a different kernel per sample, as the code does with a loop over the batch.
• Zero-valued O. In code, the middle-valued channel of O is always 0 and the other two are ±1, so O carries only the ordering of the extremes ciconv2d.py.
• Training noise only. Noise is added to the prior input during training, not at test time coco_dataset.py test.py. The paper's claim of robustness to noisy dark inputs §2.3 is not tested here.
• Decoder dependency. The bypass decoder is needed at inference, and its checkpoint is a separate file. The released test script loads it from ./checkpoints/; without it the output keeps the SD decoder's distortion test.py tab. 2.
• Hidden cost. The full model has 1.3G parameters. The released test script reports about 13 GB of GPU memory on an RTX 4090 at float16 README.md. Training used 2 GPUs with 4 samples each README.md.
• Licensing of code. There is no LICENSE file. Treat the code as all rights reserved; do not copy snippets. The repository also contains third-party code (ControlNet, CIConv, LDM, MiDaS), each with its own terms, which the repository does not restate.

Lineage

• Builds on: Retinex theory §2.1 (cited as a special case of Kubelka–Munk §2.2); Kubelka–Munk light transfer and Gaussian colour models §2.2; CIConv for learnable colour invariants §2.2 §3.3 (arXiv id not given in this card); Stable Diffusion v1.5, ControlNet, DDPM and DPM-Solver §2.3; Restormer blocks in the lightweight model §2.3.
• Compared against (zero-reference): Zero-DCE, Zero-DCE++, RUAS and SCI §1 tab. 1. (unsupervised): EnlightenGAN, PairLIE, NeRCo, CLIP-LIT, ExCNet tab. 1. (supervised): Retinex-Net, KinD, KinD++, URetinex-Net, Retinexformer, DiffLL tab. 1.
• Alternative generator tested: SR3, which shows colour bias and noise §3.3 fig. 9.
• Superseded by: not known from content already in this wiki.