wikillm

Paper · 2024-07-14 · cs.CV

Restoring Images in Adverse Weather Conditions via Histogram Transformer

Histoformer sorts pixels by intensity into bins and applies self-attention within or across bins, with a dynamic-range convolution, a dual-scale gated feed-forward and a Pearson correlation loss. One model covers snow, rain and raindrops. Author-reported: 37.41 dB PSNR on Snow100K-S, 32.08 dB on Outdoor-Rain, 33.06 dB on RainDrop.

Image enhancementSelf-attentionLayer normalizationDilated convolutionimage-restorationadverse-weatherhistogram-attentiontransformerdesnowingderaining

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Snow, rain and haze lay a similar kind of damage over a photo. Histoformer tries to remove that damage by finding pixels that look alike, even when they sit far apart in the picture. It sorts the pixels by brightness into groups. Attention then works between the groups as well as inside each group. The idea is that weather damage tends to show up in similar brightness patterns, so the model does not have to rely only on neighbouring pixels.

The network adds three smaller pieces. One is a convolution that runs over the sorted pixels. Another is a feed-forward block that mixes several filter sizes. The last is a training term that checks whether the restored image is linearly correlated with the clean one, on top of the usual pixel difference. A single network is trained for snow, rain and raindrops. The authors report that it scores best on the benchmarks they used.

TL;DR

Histoformer is an image-restoration transformer for snow, rain streaks, haze and raindrops, trained as one model for all of them §4.1. Its histogram self-attention sorts the pixels of each channel by value and attends within or across bins, so it can link far-apart pixels with similar degradation §3.2.1 eq. 4 eq. 5. Training adds a Pearson correlation loss to the L1 loss §3.3 eq. 10. Author-reported, it has the best PSNR on all four test sets and the best SSIM on three of them, not Snow100K-L tab. 1.

Problem

Transformer restoration methods save memory by limiting self-attention to a fixed spatial range or to the channel axis, which leaves long-range spatial links unmodelled §1. Weather-induced degradation often shows common patterns across pixels §1 fig. 1. Pixel-wise losses such as L1 measure closeness per pixel and do not capture linear association between output and ground truth §3.3.

Method

Overall architecture §3.1 fig. 2

  • A 3×3 convolution embeds the low-quality input with overlapping patches §3.1.
  • The network is an encoder-decoder with four levels. Histogram Transformer Blocks (HTBs) sit in every encoder and decoder level. Skip connections join encoder and decoder features at the same level §3.1.
  • Pixel-unshuffle and pixel-shuffle do the down- and up-sampling between levels §3.1.
  • A "crude" skip path carries average-pooled input features into the encoder from the second level on. It uses average pooling, a pixel-wise convolution and a depth-wise convolution §3.1.

Histogram Transformer Block §3.2 eq. 1 eq. 2

Each HTB applies two pre-norm residual steps. Layer normalisation comes before each module, and the module output is added back to the input §3.2 eq. 1 eq. 2. The first module is dynamic-range histogram self-attention (DHSA), and the second is the dual-scale gated feed-forward (DGFF).

Dynamic-range histogram self-attention (DHSA) §3.2.1 eq. 3 eq. 4 eq. 5

  1. Dynamic-range convolution eq. 3. The channels are split into two halves. The first half is sorted horizontally, then vertically, and the halves are concatenated again. A 1×1 convolution and a 3×3 depth-wise convolution follow. The sort brings pixels of similar intensity to regular positions, so the convolution can combine pixels that are far apart in the image §3.2.1.
  2. Value sort eq. 4. The convolution output is split into one value tensor and two query-key pairs. Each channel's values are sorted by intensity, and the query and key tensors are reordered with the same permutation §3.2.1.
  3. Histogram reshaping. The sorted sequence is reshaped in two ways. In bin-wise histogram reshaping (BHR), there are B bins of HW/B pixels each, which captures large-scale information. In frequency-wise histogram reshaping (FHR), there are HW/B bins of B pixels each, which captures fine detail. Every bin has the same size, for parallel computation §3.2.1.
  4. Attention and fusion eq. 5. Each reshaping has its own attention, softmax(QKᵀ/√k)·V with k the number of heads. The two outputs are multiplied element-wise §3.2.1 eq. 5.
  5. Restore order. The reordered features are sorted back to their original positions before the final 1×1 convolution §3.2.1.

Dual-scale gated feed-forward (DGFF) §3.2.2 eq. 6

A 1×1 convolution widens the channels by the factor r. The result is shuffled, then split into two branches. A 5×5 depth-wise convolution runs on one branch, and a dilated 3×3 depth-wise convolution runs on the other. The Mish activation of the second branch gates the first. The output is unshuffled and passed through a 1×1 convolution back to the input width §3.2.2 eq. 6.

Loss §3.3 eq. 7 eq. 8 eq. 9 eq. 10

The reconstruction term is the L1 norm of the difference between restored and ground-truth images §3.3 eq. 7. The correlation term is the Pearson coefficient between the two images, computed over all 3HW values §3.3 eq. 8. It is turned into a loss that is 0 when the coefficient is 1 §3.3 eq. 9. The total is the reconstruction term plus α times the correlation loss, with α = 1 by default §3.3 eq. 10 §4.3.

\rho\left(I^{hq}, I^{gt}\right) = \frac{\sum_{i=1}^{3HW}\left(I^{hq}_{i}-\overline{I}^{hq}\right)\left(I^{gt}_{i}-\overline{I}^{gt}\right)}{3HW\,\sigma\left(I^{hq}\right)\sigma\left(I^{gt}\right)} \qquad [\text{eq. 8}]

\mathcal{L} = \mathcal{L}_{rec} + \alpha\,\mathcal{L}_{cor} \qquad [\text{eq. 10}]

The pseudocode below follows the paper's equations. Points where the released code differs, or where the paper is silent, are in Paper vs code and Pitfalls.

Pseudocode
# Histogram transformer block  [§3.2] [eq. 1] [eq. 2]
# Input: features (B, C, H, W); output: features (B, C, H, W)

def histogram_transformer_block(features):
    # Pre-norm residual around histogram attention  [§3.2] [eq. 1]
    features = features + dhsa(layer_norm(features))
    # Pre-norm residual around the dual-scale feed-forward  [§3.2] [eq. 2]
    features = features + dgff(layer_norm(features))
    return features
Pseudocode
# Dynamic-range histogram self-attention  [§3.2.1] [eq. 3] to [eq. 5]
# num_bins = B; num_heads = k (the paper's k)

def dhsa(features, num_bins, num_heads):
    # Split channels; sort the first half horizontally, then vertically
    # [§3.2.1] [eq. 3]
    first, second = split_channels(features)
    first, reorder = sort_horizontal_then_vertical(first)
    reordered = concat_channels(first, second)
    # 1x1 conv, then 3x3 depth-wise conv  [§3.2.1] [eq. 3]
    mixed = depthwise3x3(conv1x1(reordered))
    # One value tensor and two query-key pairs from a channel split  [§3.2.1]
    values, queries_a, keys_a, queries_b, keys_b = split_into_five(mixed)
    # Sort each channel's values; reorder the query-key pairs the same way
    # [§3.2.1] [eq. 4]
    order = argsort_per_channel(values)
    values = gather(values, order)
    queries_a = gather(queries_a, order)
    keys_a = gather(keys_a, order)
    queries_b = gather(queries_b, order)
    keys_b = gather(keys_b, order)
    # Bin-wise reshape: B bins of HW/B pixels each (BHR)  [§3.2.1] [eq. 5]
    out_bin = attention(
        bin_wise(queries_a, num_bins),
        bin_wise(keys_a, num_bins),
        bin_wise(values, num_bins),
        num_heads,
    )
    # Frequency-wise reshape: HW/B bins of B pixels each (FHR)  [§3.2.1] [eq. 5]
    out_freq = attention(
        freq_wise(queries_b, num_bins),
        freq_wise(keys_b, num_bins),
        freq_wise(values, num_bins),
        num_heads,
    )
    # Fuse the two branches element-wise, then undo both sorts  [§3.2.1] [eq. 5]
    out = unsort(out_bin * out_freq, order)
    out = unsort_first_half(out, reorder)
    return conv1x1(out)
Pseudocode
# Scaled attention applied to each reshaped branch  [§3.2.1] [eq. 5]
# Which axis is the token axis is not stated; see Pitfalls.

def attention(queries, keys, values, num_heads):
    # Scale by 1/sqrt(k), k = number of heads, as the paper writes it  [§3.2.1]
    scores = queries @ transpose(keys) / sqrt(num_heads)
    return softmax(scores) @ values
Pseudocode
# Dual-scale gated feed-forward  [§3.2.2] [eq. 6]
# Input: features (B, C, H, W); expansion = r

def dgff(features, expansion):
    # 1x1 conv widens by r, then pixel shuffle  [§3.2.2] [eq. 6]
    widened = pixel_shuffle(conv1x1(features, expansion))
    # Split into two branches  [§3.2.2] [eq. 6]
    branch_a, branch_b = split_channels(widened)
    # 5x5 depth-wise on A; dilated 3x3 depth-wise on B  [§3.2.2] [eq. 6]
    branch_a = depthwise_conv(branch_a, kernel=5)
    branch_b = depthwise_conv(branch_b, kernel=3, dilated=True)
    # Mish of B gates A; unshuffle, then 1x1 conv to C  [§3.2.2] [eq. 6]
    gated = mish(branch_b) * branch_a
    return conv1x1(pixel_unshuffle(gated))
Pseudocode
# Training loss  [§3.3] [eq. 7] to [eq. 10]
# Inputs: restored, target (B, 3, H, W); alpha = weight of correlation term

def training_loss(restored, target, alpha):
    # Pixel term: L1 difference, averaged over elements in the code
    # [§3.3] [eq. 7]
    loss_rec = l1_difference(restored, target)
    # Per-image Pearson coefficient over all 3HW values  [§3.3] [eq. 8]
    rho = pearson_per_image(restored, target)    # (B,), in [-1, 1]
    # Loss is 0 at perfect correlation and 1 at -1  [§3.3] [eq. 9]
    loss_cor = (1 - rho) / 2
    # Weighted sum; alpha defaults to 1  [§3.3] [eq. 10] [§4.3]
    return loss_rec + alpha * mean(loss_cor)
Pseudocode
# Pearson coefficient per image pair  [§3.3] [eq. 8]

def pearson_per_image(restored, target):
    # Flatten each image to one vector over all its values  [§3.3] [eq. 8]
    x = flatten_per_sample(restored)    # (B, 3*H*W)
    y = flatten_per_sample(target)      # (B, 3*H*W)
    x = x - mean(x, dim=1)
    y = y - mean(y, dim=1)
    cross = sum(x * y, dim=1)
    # Product of the two norms; 1e-6 guards against 0/0
    # [code: basicsr/models/image_restoration_model.py#L132]
    denominator = sqrt(sum(x * x, dim=1)) * sqrt(sum(y * y, dim=1))
    return cross / (denominator + 1e-6)

Hyperparameters

NameValueWhereNotes
Training iterations300,000§4.1code: total_iter 300000 Allweather_Histoformer.yml
Blocks per level L1–L4{4, 4, 6, 8}§4.1matches config Allweather_Histoformer.yml; decoder reuses the same counts histoformer_arch.py
Base channels C36§4.1doubles at each level histoformer_arch.py
DGFF expansion r2.667§4.1config value Allweather_Histoformer.yml
Heads per level{1, 2, 4, 8}§4.1config Allweather_Histoformer.yml
Bins Bnot stated§3.2.1code sets B = number of heads; see Paper vs code
Ablation default C×B36 at level 1tab. 444 ran out of memory tab. 4
Initial batch size8§4.1per GPU in code, 4 GPUs Allweather_Histoformer.yml
Patch size128 initially§4.1code grows it to 362 over training Allweather_Histoformer.yml
OptimiserAdamW§4.1betas (0.9, 0.999), weight decay 1e-4 appear in code only Allweather_Histoformer.yml
Learning rate3e-4, then cosine to 1e-6§4.1first 92,000 iterations at 3e-4, the remaining 208,000 cosine Allweather_Histoformer.yml
Warm-upnot stated§4.1code: none Allweather_Histoformer.yml
Gradient clippingnot stated—code clips the global norm at 0.01 image_restoration_model.py
Augmentationhorizontal and vertical flips§4.1code enables geometric augmentation Allweather_Histoformer.yml
Correlation weight α1§4.3 tab. 6code hard-codes weight 1 image_restoration_model.py
HardwareNVIDIA Tesla V100§4.1text says "a" GPU; code launches 4 GPUs train.sh
Dilation of dilated depth-wise convnot stated§3.2.2dilation 2 in code histoformer_arch.py

Results

Author-reported. Setup: one model trained on the Snow100K, RainDrop and Outdoor-Rain training sets, compared with single-task networks fine-tuned per dataset and with all-in-one networks §4.1 §4.2. Methods marked * have no available source code, per the table caption tab. 1. "—" means the paper does not report that method on that dataset. Bold in the paper: Histoformer on every PSNR and SSIM except Snow100K-L SSIM, where AWRCP* is bold at 0.9341 tab. 1.

MethodSnow-S PSNRSnow-S SSIMSnow-L PSNRSnow-L SSIMOutdoor-Rain PSNROutdoor-Rain SSIMRainDrop PSNRRainDrop SSIM
SPANet29.920.826023.700.7930————
JSTASR31.400.901225.320.8076————
RESCAN31.510.903226.080.8108————
DesnowNet32.330.950027.170.8983————
DDMSNet34.340.944528.850.8772————
CycleGAN————17.620.6560——
pix2pix————19.090.710028.020.8547
HRGAN————21.560.8550——
PCNet————26.190.9015——
MPRNet————28.030.9192——
DuRN——————31.240.9259
RaindropAttn——————31.440.9263
AttentiveGAN——————31.590.9170
IDT——————31.870.9313
MAXIM——————31.870.9352
NAFNet34.790.949730.060.901729.590.9027——
Restormer36.010.957930.360.906830.030.921532.180.9408
All-in-One*——28.330.882024.710.898031.120.9268
TransWeather32.510.934129.310.887928.830.900030.170.9157
Chen et al.34.420.946930.220.907129.270.914731.810.9309
WGWSNet34.310.946030.160.900729.320.920732.380.9378
WeatherDiff₆₄35.830.956630.090.904129.640.931230.710.9312
WeatherDiff₁₂₈35.020.951629.580.894129.720.921629.660.9225
AWRCP*36.920.965231.920.934131.390.932931.930.9314
Histoformer (ours)37.410.965632.160.926132.080.938933.060.9441

Source: tab. 1.

Qualitatively, the paper says the method removes fine and large snow spots more completely than WeatherDiff, which leaves residual snow, and that it handles haze and rain marks better than the compared methods §4.2 fig. 3 fig. 4 fig. 5 fig. 6. These statements are visual and not scored.

Ablations (Outdoor-Rain, author-reported) §4.3

Dynamic-range convolution tab. 2:

VariantPSNRSSIM
Vanilla convolution31.940.9377
Vertical then horizontal sort32.030.9385
Horizontal then vertical sort (used)32.080.9389

Self-attention design tab. 3:

VariantPSNRSSIM
MDTA30.940.9278
TKSA31.120.9295
DHSA without BHR31.050.9301
DHSA without FHR31.790.9364
DHSA (full)32.080.9389

C×B, the attention size, at level 1 tab. 4:

C×BPSNRSSIM
1230.430.9283
2031.560.9312
2831.940.9379
3632.080.9389
44out of memoryout of memory

Feed-forward design tab. 5:

VariantPSNRSSIM
FN31.320.9313
GDFN31.420.9347
DANB31.560.9351
MSFN31.780.9367
DGFF32.080.9389

Correlation loss weight α tab. 6:

SettingPSNRSSIM
Without L_cor (α = 0)31.770.9358
α = 0.132.010.9369
α = 1 (default)32.080.9389
α = 532.030.9392
α = 1031.960.9375

The paper reports that the sort order barely matters (0.05 dB) and that the loss weight has little influence §4.3.

Paper vs code

Official code: the abstract links github.com/sunshangquan/Histoformer. The owner matches the first author's name. Checked at commit 1f045f06c03551c31504d8042dbe6ff9b9569108. The repository has no LICENSE file, so no code is quoted here.

  1. Attention normalisation. The paper scales scores by 1/√k, where k is the number of heads, and uses a softmax §3.2.1 eq. 5. The code L2-normalises queries and keys, multiplies scores by a learnable per-head temperature that starts at 1, and uses a softmax with +1 in the denominator. It has no 1/√k factor histoformer_arch.py histoformer_arch.py histoformer_arch.py.
  2. Bin count. The paper treats B as a free parameter and gives no default §3.2.1. The code sets the bin count to the number of heads, so it is 1, 2, 4 and 8 across the four levels histoformer_arch.py Allweather_Histoformer.yml.
  3. Sort order. Equation 3 writes Sort_v(Sort_h(·)), horizontal first §3.2.1 eq. 3. The code sorts along height first, then width histoformer_arch.py. Table 2 gives 32.08 for "Horiz.+Verti." and 32.03 for "Verti.+Horiz." tab. 2. The text does not define the labels, so it is unclear which order the table's best result used.
  4. Un-sort point. The paper un-sorts before the final 1×1 convolution §3.2.1. The code un-sorts after it histoformer_arch.py. This should be equivalent, because a 1×1 convolution acts on each pixel separately; this equivalence is our reading, not stated in the paper.
  5. Layer normalisation. The paper says LN eq. 1 eq. 2. The code's normalisation has no learnable scale or shift: those parameters are commented out histoformer_arch.py histoformer_arch.py.
  6. DGFF resolution. The paper gives the shuffle operations with no scale §3.2.2 eq. 6. The code applies a 2× pixel shuffle before the depth-wise convolutions and a 2× unshuffle after them, so the 5×5 and dilated convolutions run on a grid twice as large histoformer_arch.py histoformer_arch.py histoformer_arch.py.
  7. Crude skip path. The compiled paper starts the crude skip after the first stage §3.1. A commented-out passage in the TeX source (not compiled) says the first crude feature is the low-quality input. The code feeds the raw input through pooling, a 1×1 convolution and a depth-wise convolution into level 2, and chains the path to levels 3 and 4. Each junction uses a 1×1 convolution that the paper does not describe histoformer_arch.py histoformer_arch.py.
  8. Refinement blocks. The paper lists only the four encoder and decoder levels §4.1. The code adds four refinement blocks at the finest decoder level Allweather_Histoformer.yml histoformer_arch.py histoformer_arch.py.
  9. Global residual. The code adds the input image to the network output histoformer_arch.py. The paper does not describe this.
  10. Reconstruction term. The paper writes the L1 norm without normalisation §3.3 eq. 7. The config and the loss use L1 with mean reduction, which divides by the number of elements Allweather_Histoformer.yml loss_util.py losses.py. The paper does not say which convention it used, and the relative weight of the two terms depends on it.
  11. Correlation extent. Section 3.3 says the loss targets "patch-level" correlations §3.3, but equation 8 sums over the whole image. The code does the same, per image over all channels and pixels, then averages over the batch image_restoration_model.py image_restoration_model.py.
  12. Non-finite entries. The code drops NaN and infinite correlation values from the average image_restoration_model.py. The paper does not mention this.
  13. Correlation term weight. The code adds the correlation term with weight 1 and no α image_restoration_model.py image_restoration_model.py image_restoration_model.py.
  14. Gradient clipping. The code clips the global gradient norm to 0.01 at every step image_restoration_model.py. The paper does not mention it.
  15. Training schedule. The paper gives batch size 8 and initial patch size 128, "akin to" progressive learning §4.1. The released config is progressive: per-GPU batch sizes [8, 5, 2, 1, 1] over patch sizes [128, 160, 256, 320, 362], with iterations [92000, 84000, 56000, 36000, 32000]. A fixed 128 setting is commented out Allweather_Histoformer.yml. The paper does not state the progression.
  16. Model defaults. The class defaults differ from the config: class-level block counts are [4, 6, 6, 8] and the expansion factor is 2.66 histoformer_arch.py. The config overrides them with [4, 4, 6, 8] and 2.667 Allweather_Histoformer.yml. The paper matches the config.
  17. Metric colour space. The paper does not say. The evaluation script and the config compute PSNR and SSIM on the Y channel of YCbCr compute_psnr.py Allweather_Histoformer.yml.

Pitfalls

  • Bins and attention layout. The paper does not define the token axis of the attention. Read from the code's reshape strings and matrix product, the score matrix is square over channels times bins, per head, and each score sums over the pixels inside a bin. This is our reading of the code, not a stated fact histoformer_arch.py histoformer_arch.py. At level 1 with one head, BHR and FHR produce the same layout, because the bin count is 1. Level 1's C×B = 36 default is consistent with this tab. 4.
  • Sorting cost and gradients. Each forward pass sorts every channel's values across all positions, then gathers and scatters. The permutation itself carries no gradient, and tie order is not stated histoformer_arch.py.
  • In-place write. The code writes the sorted first half back into its input tensor histoformer_arch.py. Copy the input before reusing it.
  • Memory limit. Level-1 C×B = 44 ran out of memory on the paper's hardware tab. 4 §4.3.
  • Correlation loss is blind to affine intensity changes. The Pearson coefficient is unchanged when the restored image is a positive scalar times the clean image plus a constant. The correlation term therefore does not penalise a global brightness or contrast change; only the L1 term does. This follows from the formula and is not discussed in the paper.
  • Flat images. Near-constant images have a near-zero denominator. The code adds 1e-6 image_restoration_model.py. The paper does not mention it.
  • Coefficient normalisation in the TeX source. A commented-out definition of σ in the TeX source is an unnormalised sum of squares. With that definition, equation 8 would be off by a factor of 3HW. The code computes the standard coefficient. Follow the code, or the standard definition.
  • Batch size is per GPU in the config. Four GPUs with a per-GPU batch of 8 means an effective batch of 32 in the released config. The paper's text says batch size 8 and mentions one V100 §4.1 Allweather_Histoformer.yml train.sh.
  • Hyperparameters in the supplement. Section 4.1 refers readers to the supplement for more training details §4.1. The supplement source (sec/X_suppl.tex) is not included by main.tex, so this card does not use it.
  • Tables not compiled. The source contains parameter-count, FLOPs, runtime and NIQE tables under tab_sec/, but main.tex does not include them. This card does not report them.
  • Test-time padding. The test script reflect-pads inputs to a multiple of 8 and crops the output back test_histoformer.py test_histoformer.py. The paper does not mention padding.
  • Real-world checkpoint. The README lists a separate real-world checkpoint in addition to the all-weather one README.md. The paper does not say how that model was trained.

Lineage

  • Compared against: the baselines in tab. 1.
  • Ablation alternatives: MDTA and TKSA attention tab. 3; FN, GDFN, DANB and MSFN feed-forward modules tab. 5.
  • Builds on: not stated beyond the cited baselines and ablation alternatives.
  • Superseded by: not stated in the paper or in this wiki.

Source and license

Adapted from “Restoring Images in Adverse Weather Conditions via Histogram Transformer” by Shangquan Sun, Wenqi Ren, Xinwei Gao, Rui Wang, Xiaochun Cao (arXiv:2407.10172), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/sunshangquan/Histoformer at commit 1f045f06c035 (No license file in the repository (unlicensed; no code quoted)). Any quoted code is shown under that license.