Image blending

Overview

Image blending places a source object into a target background so that the result reads as one photo. The difficulty is the boundary: brightness, colour and texture on either side of the mask edge should agree, while the object's own fine detail should survive the edit 1703.07195 §1. Gradient-domain methods keep the high-frequency detail but can leave colours that look wrong. Learned colour models can produce realistic colour, but direct high-resolution generation is costly 1703.07195 §1 1703.07195 §3.4.

The idea that the variants share is an output at the target resolution, with fine structure taken from the source inside the mask and the background outside it. Colour is the part that may come from a different source.

Variants

Gradient-domain blending (classical baseline)

These methods solve for an image whose gradients follow the source inside the mask and the background outside it 1703.07195 §3.4. The paper's comparison set consists of Poisson image editing, modified Poisson editing and multi-splines blending 1703.07195 §4.3. This page does not summarise those methods beyond that; they have no card here.

Learned colour plus Gaussian-Poisson solve (GP-GAN)

Card: GP-GAN: Towards Realistic High-Resolution Image Blending.

• Phase one, colour estimate. A supervised encoder-decoder, the Blending GAN, maps the composite to a 64 × 64 colour image. It is trained with 0.999 weight on an L2 term and 0.001 on an adversarial term 1703.07195 eq. 2 1703.07195 §3.3. Training pairs are two frames from the same fixed webcam, with the destination frame standing in for the ground truth 1703.07195 §4.1 1703.07195 §3.3.
• Phase two, full-resolution solve. A closed-form Gaussian-Poisson objective (eq. 9) is solved at each level of a Laplacian pyramid, from coarse to fine. The colour target at the coarsest level is the network output, and each level passes its result on as the next target 1703.07195 §3.4 1703.07195 alg. 1.
• Reported result. On 500 images, the paper reports a realism score of −0.069 against −0.140 for the best baseline (multi-splines blending) 1703.07195 tab. 1. In a crowd-sourced comparison GP-GAN received 947 votes against 770 for that baseline 1703.07195 tab. 2.

When to use which

The paper tests one benchmark only, so the table below states constraints and the evidence the paper gives for each.

Implementation sketch

GP-GAN's two phases are the common core for that variant. Only the first phase is learned; the second is a closed-form solve.

# Two-phase blending  [arxiv:1703.07195 §3.3-§3.4] [alg. 1]
# Input:  source, background (H, W, 3) in [0, 1]; mask (H, W) in {0, 1}
# Output: blended (H, W, 3)

def blend(source, background, mask, G):
    # Phase one: learned low-resolution colour estimate  [§3.3] [eq. 2]
    small = composite_at_64px(source, background, mask)
    colour = G(small)

    # Phase two: coarse-to-fine closed-form solves on a Laplacian pyramid  [§3.4]
    for level in levels_coarse_to_fine(H, W):
        # Bring the previous estimate to this level's size  [alg. 1]
        colour = resize(colour, size_at(level))
        # Gradients: source inside the mask, background outside  [eq. 8]
        guidance = select_gradients(source[level], background[level], mask[level])
        # Solve eq. 9 at this level: gradient match plus colour match  [eq. 9]
        colour = solve_gaussian_poisson(guidance, colour, beta=1.0)
    return colour

Notes on the parts that differ between implementations:

• The colour weight is written as β in eq. 5 and as λ in eq. 9. The paper appears to use one weight under two symbols, and sets it to 1 1703.07195 §3.4 1703.07195 eq. 5 1703.07195 eq. 9.
• The closed form follows Frankot and Chellappa 1703.07195 §3.4. The paper gives the discrete objective but not the transform basis or the Gaussian kernel size.
• The Blending GAN's encoder width is not stated in the paper. Its layer widths appear only in its figures 1703.07195 fig. 3 1703.07195 fig. 4.

Hyperparameters stated in the paper

Not stated in the paper: the critic's learning rate and update schedule, the encoder bottleneck width, the Gaussian σ in eq. 7 and eq. 9, the Laplacian smoothing, and the number of pyramid levels 1703.07195 §3.3 1703.07195 §3.4.

Contradictions

• Wording against the table. The paper says most users preferred GP-GAN 1703.07195 §4.4. In its vote table, GP-GAN has 947 of 2,979 votes, about 32%, and the table does not show a majority over the other methods 1703.07195 tab. 2. Use the table's numbers, not the sentence.
• Other papers. None known. This page has one paper, and its benchmark results have no independent replication on this page.

Papers

• GP-GAN: Towards Realistic High-Resolution Image Blending: a supervised low-resolution GAN colour estimate, followed by a multi-scale closed-form Gaussian-Poisson solve.