Fast End-to-End Trainable Guided Filter

In plain words

Many image tools, such as smoothing, retouching or depth estimation, need a network that outputs a full-size picture. Running a big network at full size is slow and memory-hungry. The usual trick is to shrink the input, run the network on the small picture, and then blow the result back up. The hard part is the blow-up: a plain resize gives blurry edges.

This paper replaces the final blow-up with a "guided filter". The filter takes the small output and a full-size picture of the scene, and uses the full-size picture to decide where edges should be. The authors rewrote this filter as a set of ordinary neural-network operations, so it can be trained together with the network. The network can then learn what the blow-up should look like for each task, and the guide picture itself can be learned too.

The result is a general framework, called DGF, for many pixel-by-pixel image tasks. The authors report that it is much faster than the comparison methods at high resolution while keeping or improving quality.

TL;DR

• DGF = a small-resolution CNN + a differentiable, learnable guided filtering layer that upsamples its output to full resolution §4.
• The layer is trained end to end; a learned 1x1-convolution block can produce the guidance map §3.4.
• Author-reported: on five MIT-Adobe FiveK retouching-style operators it beats DBL on all five and CAN on style transfer, dehazing and retouching (CAN is higher on L0 smoothing and detail manipulation), and runs 10-100x faster §5.4 tab. 2.
• The official repo ships inference code and pretrained weights for the five image tasks; the training scripts are on a different branch.

Problem

Fully convolutional networks (FCNs) for dense prediction are costly on large inputs in compute and memory §1. Downsampling and then upsampling the output loses detail and edges unless the upsampling is guided by the high-resolution image §1. Existing guided and bilateral upsamplers are either hand-designed, not trainable with the network, or slow §2.1. A plain guided filter is differentiable in theory but was used only as post-processing; it cannot be trained with the FCN §3.3.

Method

Guided filter, as used here. Given a low-resolution input I_l, a high-resolution input I_h and a low-resolution network output O_l, the guided filter fits, for every local window ω_k, a linear model between the guide and the output eq. 1. The coefficients A_l and b_l are then upsampled to full resolution and applied to the high-resolution guide eq. 2 §3.2. The window means come from a box (mean) filter with radius r; ε regularises the division §3.3.

1. Make the layer differentiable §3.3 alg. 1. The box filter is an ordinary mean filter, the coefficients are simple products and divisions, and the bilinear upsampling is differentiable. The paper gives the gradients to back-propagate through the layer alg. 1. The official code instead relies on automatic differentiation of the forward pass; it has no hand-written backward guided_filter.py.
2. Learnable convolutional form §3.5 tab. 1 fig. 4. The mean filter is replaced by dilated 3x3 convolutions (dilation = r), and the local linear model by a block of 1x1 convolutions that takes the local covariance and variance as input. The ε term is removed. The parameters are then learned.
3. Learnable guidance map F(I) §3.4. A two-layer block (1x1 convolution, adaptive normalisation, leaky ReLU, 1x1 convolution) turns I_h and I_l into guidance maps G_h and G_l. The first layer has 16 channels by default §3.4.
4. FCN on the small image §4.1. The backbone C_l processes I_l and produces O_l. The paper uses a context aggregation network (CAN) in the image-processing experiments §5.2. Any FCN can be dropped in §4.1.
5. Full pipeline §4 fig. 3. Downsample the input, run the FCN, then run the guided filtering layer with (I_l, I_h, O_l) to get the full-resolution output. The whole model trains from scratch with a single loss §4.3.
6. Four variants §4.2. DGF_s: original guided filter as post-processing on a separately trained FCN. DGF_b: learnable layer with identity guidance. DGF_b^c: convolutional layer. DGF^c: convolutional layer plus learned F(I).

# Guided filtering layer, differentiable  [§3.2] [§3.3] [eq. 1] [eq. 2]
# Inputs:  guide_lr, target_lr (B, C, h, w); guide_hr (B, C, H, W)
# Output:  output_hr (B, C, H, W)

def guided_filter_layer(guide_lr, target_lr, guide_hr, radius, eps):
    # Local window means via the mean filter f_mu  [§3.3]
    mean_guide = box_mean(guide_lr, radius)
    mean_target = box_mean(target_lr, radius)
    # Local variance and covariance inside each window  [§3.2]
    var_guide = box_mean(guide_lr * guide_lr, radius) - mean_guide ** 2
    cov_guide_target = (
        box_mean(guide_lr * target_lr, radius)
        - mean_guide * mean_target
    )
    # Per-window linear coefficients, the local model of [eq. 1]  [§3.2]
    coef_a = cov_guide_target / (var_guide + eps)
    coef_b = mean_target - coef_a * mean_guide
    # Bilinear upsampling of A and b to the high-resolution grid  [§3.3]
    coef_a_hr = upsample_bilinear(coef_a, guide_hr.shape[2:])
    coef_b_hr = upsample_bilinear(coef_b, guide_hr.shape[2:])
    # Linear model applied to the high-resolution guide  [eq. 2]
    return coef_a_hr * guide_hr + coef_b_hr

# Deep guided filtering network, forward pass  [§4] [fig. 3]
# Input:  image_hr (B, 3, H, W) in [0, 1]
# Output: output_hr (B, 3, H, W)

def dgf_forward(image_hr, low_size=64):
    # Shrink the input so its shorter side is low_size  [§5.3]
    image_lr = resize_to_short_side(image_hr, low_size)
    # Run the FCN on the small image; output has 3 channels  [§4.1] [tab. 1]
    output_lr = fcn(image_lr)
    # Guidance: identity for DGF_b, learned F(I) for DGF  [§3.4] [§4.2]
    guide_lr = guidance(image_lr)
    guide_hr = guidance(image_hr)
    # Guided filtering layer restores full resolution  [§4] [eq. 2]
    return guided_filter_layer(guide_lr, output_lr, guide_hr)

Notes on the pseudocode: box_mean is a sum over the (2r+1) window divided by the number of pixels inside the image, which keeps the border windows normalised §3.2 guided_filter.py. guidance(...) is the identity for DGF_b and a learned block for DGF; the convolutional variants use dilated convolutions for box_mean and a learned 1x1 block for the coefficients §3.5.

Layer shapes used in the image tasks. The FCN C_l is a context aggregation network: 24 channels, leaky ReLU slope 0.2, adaptive normalisation between layers, dilations that grow exponentially tab. 1 §5.2. The exact layer list is in the Hyperparameters section below.

Task-specific use of the layer. For depth, saliency and segmentation the high-resolution input is not downsampled; the layer is applied at full resolution to the network's own output §6.2 dss.py.

Hyperparameters

Image-processing experiments §5.2-§5.3 tab. 1:

Architecture of C_l(I_l) and F(I) tab. 1. Kernel, channels and dilation as printed; "AdaptNorm" = adaptive normalisation; "Bias" as printed.

Computer-vision experiments §6.2 tab. 5:

Computer-vision data §6.1: KITTI, 29,000 training and 1,159 evaluation pairs from the official training set, 200 test images §6.1; MSRA-B split from §6.1; PASCAL VOC 2012 with 10,582 augmented training images and 1,449 validation images §6.1.

Results

Author-reported. Setup: MIT-Adobe FiveK test set, input and target resized to 1024 short side, metrics MSE, PSNR and SSIM §5.4.2. Baselines: CAN and DBL, trained with their official procedures §5.3; DJF also appears §5.4.2. The paper's rank colours (red, green, blue) in tab. 2 are not reproduced here.

tab. 2 Quantitative comparison, image-processing tasks:

Headline numbers §5.4.2: DGF^c reaches 26.17 dB PSNR on style transfer, 4.86 dB above CAN and 2.85 dB above DBL. The paper states DGF outperforms DBL on all five tasks and all three metrics §5.4.2, and DJF on all tasks §5.4.3. Against CAN, the paper's own table shows CAN ahead on L0 smoothing and detail manipulation tab. 2.

tab. 3 Upper bound, with the low-resolution ground truth replacing O_l §5.4.3:

tab. 4 Speed and memory against the resolution of I_l, on a Titan X GPU §5.4.5 §5.4.1:

Running time and memory §5.4.1 fig. 5: at 2048 x 2048, DGF_b, DGF_b^c and DGF^c take 6, 6 and 21 ms. CAN takes 160 ms and DBL 51 ms at the same size. DJF takes 46 ms at 1024 x 1024 against 5 ms for DGF §5.4.3. CAN uses nearly 10 GB at 2048 x 2048; the DGF_b variants stay under 1 GB up to 3072 x 3072 §5.4.1. The paper also gives theoretical complexity classes for each method §5.4.1.

Computer-vision results tab. 6 (author-reported):

The CV results are on the paper's own splits §6.1. DGF takes 34 ms on a 512 x 512 image against 432 ms for DenseCRF §6.3.

Cross-resolution: the pretrained models, applied without fine-tuning to inputs of 512 to 2048 short side, perform about equally well on all tasks except style transfer, which the paper attributes to its dependence on resolution §5.4.4 fig. 6. Ablations on radius, I_l resolution and the dilation rate of F(I) are in fig. 7 and tab. 4; the paper reports that r = 1 gives the best PSNR and that increasing the F(I) dilation rate helps to a degree §5.4.5.

Paper vs code

Official code: https://github.com/wuhuikai/DeepGuidedFilter, owned by the first author (Huikai Wu) [README]. Two refs were checked:

• code_commit = bcebf35 on the release branch (the README says to check out release) README.md. It has inference code, pretrained weights for the five image tasks, the CV inference scripts, and the layer modules.
• master (commit 0caaf2d) has the training, test and dataset scripts and the PyTorch guided-filter package that module.py imports. Lines cited as "(master)" refer to that commit.

The repository has no LICENSE file at its root; the only LICENSE in the tree is inside ComputerVision/MonoDepth, which is third-party code and is not used here.

Discrepancies (each with both citations):

• LR network depth and dilations. Paper: seven 3x3 layers with dilations 1, 1, 2, 4, 8, 16, 1, then one 1x1 layer tab. 1. Code: build_lr_net with default layer=5 builds six 3x3 layers with dilations 1, 2, 4, 8, 16, 1, then one 1x1 layer module.py. Master training uses the same default train_lr.py.
• F(I) in the full DGF. Paper: first guidance layer has 16 channels §3.4. Code: DeepGuidedFilterAdvanced uses 15 module.py.
• F(I) kernel size. Paper text says both guidance layers are 1x1 §3.4; Table 1 gives kernel 3 and dilation 1 for the first F(I) layer tab. 1. Code default for DGF^c is dilation 0, i.e. 1x1 module.py.
• Epsilon. Paper default epsilon is 1e-8 §3.3. Code default is 1e-8 for DeepGuidedFilter module.py but 1e-4 for DeepGuidedFilterAdvanced (DGF) module.py.
• Output clamping. Code clamps the output to [0, 1] in all variants module.py, module.py, module.py. Paper does not state it.
• Gradient clipping. Paper states no clipping §5.3. Master training sets clip = 0.01 for the 512-stage and the 30-epoch stage train_hr.py and for fine-tuning train_hr_finetune.py. The clipping call is in train_base.py.
• Random training size. Paper: short side from 512 to 1672 §5.3. The generator script sets the upper bound with MAX_SIZE_TOTAL = 2048 precompute_size.py. The shipped size list was not range-checked.
• Oversized images. The 30-epoch stage skips images with more than 2048^2 pixels train_hr.py; the fine-tuning script does the same train_hr_finetune.py. Paper does not mention it.
• Downsampling method. Paper does not state how I_l is produced. Code uses nearest-neighbour resizing of the short side to 64 for both training and prediction dataset.py.
• Initialisation. Paper does not state one. Master module.py applies an identity-style initialisation to the convolutions (zero except centre taps, Xavier for the last layer) and sets BatchNorm to weight 1 and bias 0 module.py, applied at module.py and module.py. The release module omits this.
• Adaptive normalisation. Paper cites the normalisation of CAN and gives no detail §5.2. Code: learned mix of identity and batch norm (momentum 0.999, eps 0.001) module.py.
• Convolutional box filter. Paper describes learnable dilated convolutions §3.5. Code initialises the 3x3 box filter to ones and keeps it trainable guided_filter.py.
• Fine-tuning. Paper: "we further train the network" for 30 epochs §5.3. Master has a separate train_hr_finetune.py that loads the low-resolution network from a previous run before training train_hr_finetune.py. train_hr.py continues from its own 150-epoch run train_hr.py. The README offers both as alternatives: training from scratch, or low-resolution pretraining followed by fine-tuning README.md (master).
• Variant names. The README maps paper names to flags: guided_filter = DGF_s, deep_guided_filter = DGF_b, deep_guided_filter_advanced = DGF, deep_conv_guided_filter = DGF_b^c, deep_conv_guided_filter_adv = DGF^c README.md (master).
• Computer-vision hyperparameters. No discrepancies found: depth r = 4, eps = 1e-2 monodepth_simple.py; saliency r = 8, eps = 1e-2, with the guided layer also applied to the network output predict.py; segmentation r = 4, eps = 1e-2 deeplab_resnet.py. The CV guidance maps are learned 1x1 convolutions with ReLU, 3 to 64 channels, then to 1 (saliency) or to the label count (segmentation), which the paper does not state dss.py, deeplab_resnet.py.

Pitfalls

• Use the release branch for inference, master for training. The release branch has no training or dataset scripts and no guided_filter_pytorch package; module.py imports that package, so install it from GuidedFilteringLayer/ on master.
• No license. The repository has no LICENSE. Do not copy code; use the pseudocode above and cite the lines.
• Fix the layer count before you train. The paper's eight-layer table and the code's default six-layer LR net do not match tab. 1 module.py.
• Epsilon and radius are sensitive. The division by variance plus epsilon decides how strongly edges are kept. The code uses a much larger epsilon for the CV tasks (1e-2) than the default image task (1e-8) tab. 5 §3.3.
• Border normalisation. The code divides the window sums by a count of valid pixels computed from a box filter over ones, so windows at the image border are normalised by their actual size guided_filter.py. Reimplementations should keep this normalisation.
• Nearest downsampling and old tooling. The code relies on nearest-neighbour resizing, which the paper does not specify, and on transforms.Scale, which the README pins with PyTorch 1.1 and torchvision 0.2. Newer torchvision versions may not have it.
• Latent bug in random crop. RandomCrop calls img.crop with four arguments, but PIL expects one box tuple dataset.py. It runs only when fine_size > 0, which the training scripts do not set.
• Clamping and quantisation. Outputs are clamped to [0, 1] inside the model module.py, and the predict script writes 8-bit images. Values outside the range are clipped, not rescaled.
• Gradient clipping at 0.01. Training follows the paper's Adam and learning rate, but the code clips gradients at norm 0.01 train_hr.py. Reproducing the paper's numbers without this may differ.
• Speed numbers are for one setup. The timings are on a Titan X with the authors' implementation §5.4.1. The paper's 10-100x claim combines several baselines and resolutions.
• Results are author-reported. The paper compares on one dataset and one set of baselines. Style transfer depends on resolution §5.4.4.
• Shipped weights are for FiveK tasks. The release includes weights for the five image tasks; the CV weights are not in the repository release branch file list.

Lineage

• Builds on: the guided filter of He et al. §3.2 (no arXiv entry in this wiki).
• Preliminary version: Wu et al., CVPR 2018, with a fixed, non-learnable layer §1 (not in this wiki).
• Compared against: CAN (Chen et al., ICCV 2017) §2.2 tab. 2; DBL (Gharbi et al., deep bilateral learning) §5.3 tab. 2; DJF (Li et al., 2016) §5.4.2 tab. 2.
• Other joint upsamplers discussed: joint bilateral upsampling, bilateral-space methods §2.1.
• Superseded by: not known from content already in this wiki.