Guided filter

Overview

The guided filter takes a guide image and an input image, and returns an output that follows the structure of the guide. For every local window it fits a linear model between the guide and the output 1803.05619 §3.2 eq. 1. The filter comes from He et al. and is not on a card in this wiki; the paper uses it as its starting point 1803.05619 §3.2.

The problem it addresses is cost on large inputs. A fully convolutional network (FCN) is expensive at full resolution in compute and memory. Shrinking the input, running the network and then upsampling loses edges, unless the upsampling is guided by the full-resolution image 1803.05619 §1. Earlier guided and bilateral upsamplers were hand-designed, could not be trained with the network, or were slow 1803.05619 §2.1. A plain guided filter could in principle be differentiated, but it was used only after training, so the network could not learn with it 1803.05619 §3.3.

The paper's contribution is a guided-filter layer that is differentiable, and can therefore be trained end to end with a small FCN. The layer computes its coefficients on the low-resolution grid, upsamples them, and applies them to the full-resolution guide 1803.05619 §3.2 eq. 2. The learnable convolutional versions also replace the fixed mean filter and linear model with trainable layers 1803.05619 §3.5.

Variants

All four variants come from one paper, and each is named in its paper's notation 1803.05619 §4.2. Card: Fast End-to-End Trainable Guided Filter.

Post-processing guided filter (DGF_s)

The classical filter, with no learned parameters, applied after a separately trained FCN 1803.05619 §4.2. Training does not include the filter, so the network cannot adapt to it.

Learnable layer with identity guidance (DGF_b)

The differentiable layer is trained together with the FCN, using one loss 1803.05619 §4.3. The guidance is the identity, so the guide is the input image 1803.05619 §3.4 §4.2.

Convolutional layer (DGF_b^c)

The mean filter is replaced by dilated 3x3 convolutions, with the dilation set to the radius r. The local linear model becomes a block of 1x1 convolutions that reads the local covariance and variance. The ε term is removed 1803.05619 §3.5 tab. 1. Dilated convolutions are described on their own concept page.

Convolutional layer with learned guidance (DGF^c)

The convolutional layer above, plus a learned guidance map F(I). F(I) is a two-layer block: a 1x1 convolution, adaptive normalisation, a leaky ReLU and a second 1x1 convolution. Its first layer has 16 channels by default 1803.05619 §3.4. The paper's computer-vision table calls the full model DGF 1803.05619 tab. 6.

When to use which

The evidence here comes from one paper, with its own datasets and baselines. Each row below states what the paper's numbers support.

Implementation sketch

The paper gives the layer as equations and a backward pass [arxiv:1803.05619 §3.3] alg. 1. The sketch below follows those equations in our own words. The repository has no LICENSE, so do not copy its code.

# Guided filtering layer, differentiable  [§3.2] [§3.3] [eq. 1] [eq. 2]
# Inputs:  guide_lr, target_lr (B, C, h, w); guide_hr (B, C, H, W)
# Output:  output_hr (B, C, H, W)

def guided_filter_layer(guide_lr, target_lr, guide_hr, radius, eps):
    # Window means from the mean filter of radius r  [§3.3]
    mean_guide = box_mean(guide_lr, radius)
    mean_target = box_mean(target_lr, radius)
    # Local variance and covariance inside each window  [§3.2]
    var_guide = box_mean(guide_lr * guide_lr, radius) - mean_guide ** 2
    cov_gt = box_mean(guide_lr * target_lr, radius) - mean_guide * mean_target
    # Per-window linear coefficients; eps regularises the division  [§3.2]
    coef_a = cov_gt / (var_guide + eps)
    coef_b = mean_target - coef_a * mean_guide
    # Bilinear upsampling of the coefficients is differentiable  [§3.3]
    coef_a_hr = upsample_bilinear(coef_a, guide_hr.shape[2:])
    coef_b_hr = upsample_bilinear(coef_b, guide_hr.shape[2:])
    # Linear model applied to the full-resolution guide  [eq. 2]
    return coef_a_hr * guide_hr + coef_b_hr

# Deep guided filtering network, forward pass  [§4] [fig. 3]
# Input:  image_hr (B, 3, H, W) in [0, 1]
# Output: output_hr (B, 3, H, W)

def dgf_forward(image_hr, low_size=64):
    # Shrink the input so its shorter side is low_size  [§5.3]
    image_lr = resize_to_short_side(image_hr, low_size)
    # The FCN runs on the small image  [§4.1] [tab. 1]
    output_lr = fcn(image_lr)
    # Guidance is the identity (DGF_b) or a learned F(I) (DGF^c)  [§3.4]
    guide_lr = guidance(image_lr)
    guide_hr = guidance(image_hr)
    # The guided layer restores full resolution  [§4] [eq. 2]
    return guided_filter_layer(guide_lr, output_lr, guide_hr)

Settings used in the paper's image-processing experiments:

• Input I_l has a shorter side of 64 for every resolution 1803.05619 §5.3.
• Training: 150 epochs at a shorter side of 512, then 30 more epochs at a random shorter side between 512 and 1672. Adam, learning rate 0.0001, batch size 1, L2 loss 1803.05619 §5.3 §4.3.
• Radius r = 1 and ε = 1e-8 by default 1803.05619 §3.3. In the computer-vision tasks the paper uses r = 4 for depth and segmentation, r = 8 for saliency, and ε = 1e-2 1803.05619 tab. 5.
• FCN: a context aggregation network with 24 channels and leaky ReLU slope 0.2 1803.05619 §5.2 tab. 1.

The paper's text and its released code differ in several details that change results: layer count and dilations of the FCN, the first F(I) channel count, the ε default for the full model, output clamping, gradient clipping and the downsampling method. The paper card lists each difference with its code citation. Use the paper's values as the starting point, and check each difference against the card before relying on it.

Contradictions

• Speed claim. The paper's summary reports 10 to 100 times faster 1803.05619 §5.4. Its own timings at 2048 x 2048 give 51 ms for DBL and 160 ms for CAN, against 6 to 21 ms for the DGF variants 1803.05619 §5.4.1. Those figures are about 2.4 to 8.5 times for DBL and about 7.6 to 27 times for CAN. The 10 to 100 times claim is therefore not matched by these numbers against DBL, and only partly against CAN.
• Against CAN. The paper states that DGF outperforms DBL on all five image tasks 1803.05619 §5.4.2. Its table shows CAN ahead on L0 smoothing and detail manipulation, and DGF^c ahead of CAN on style transfer, dehazing and retouching 1803.05619 tab. 2.
• Saliency. On saliency, DenseCRF has a higher F-measure than DGF, 91.87% against 91.75%. DGF is ahead on segmentation, 73.58% against 72.69% 1803.05619 tab. 6.
• Inside the paper. The text says both guidance layers of F(I) are 1x1 1803.05619 §3.4. Table 1 gives the first layer a 3x3 kernel and dilation 1 1803.05619 tab. 1.
• Upper bound. The paper's upper-bound table replaces the FCN output with the low-resolution ground truth. Every DGF variant gains about 10 dB on retouching there, and the gap to the main table is the cost of the FCN output 1803.05619 tab. 2 tab. 3.

Papers

• Fast End-to-End Trainable Guided Filter: makes the guided filter a differentiable layer that can be trained end to end, with four variants (DGF) for image processing and dense prediction.