GP-GAN: Towards Realistic High-Resolution Image Blending

In plain words

When you paste an object into a photo, the seams and colours rarely match. Classic tools fix this by adjusting the pixel gradients so that the edges of the pasted part blend with their surroundings. They work at full resolution, but the colours often look wrong. Neural image generators make realistic colours, but they struggle to produce large images with fine detail.

GP-GAN combines the two. A small neural network, trained on pairs of photos of the same scene taken under different conditions, learns what a realistic blend looks like at low resolution. The result is used only for colour. A classical solver then works through the image from coarse to fine, keeping the fine edges and texture of the original pasted image while taking the colour from the network.

The authors report that this produces fewer bleeding edges and fewer artifacts than the classic methods on the paper's benchmark. In a crowd-sourced study, GP-GAN received the most votes of the methods compared.

TL;DR

• Input: a composite image (object pasted into a background), a mask, and optionally the two source images §3.1.
• Phase one: a supervised encoder-decoder "Blending GAN" makes a 64×64 realistic colour estimate §3.3.
• Phase two: a closed-form Gaussian-Poisson solve, applied at each level of a Laplacian pyramid, gives full-resolution output §3.4 alg. 1.
• Trained on Transient Attributes only; benchmark numbers are author-reported.

Problem

Classic gradient-domain blending (Poisson image editing and relatives) keeps high-frequency detail, but the colours it produces are often unrealistic, with artifacts §1. Training a GAN directly at high resolution is costly in computation and memory, and the GAN outputs that the authors tested were blurry beyond small sizes §3.4. The paper's assumption is that a low-resolution GAN output can be trusted for colour, and that the high-resolution detail should come from the input image §3.4.

Method

Notation. The composite image is x = x_src ∗ x_mask + x_dst ∗ (1 − x_mask), where ∗ is element-wise multiplication eq. 1. The goal is a blended image x̃ with the original resolution §3.1.

Blending GAN (phase one). A generator G maps the composite x to a realistic image x̃_l. It is an encoder-decoder inspired by context encoders §3.3. The encoder is a stack of convolutions with batch normalisation and leaky ReLU, reduced to a 1×1 bottleneck; the decoder mirrors it with transposed convolutions and ends in a tanh layer fig. 3 fig. 4 model.py. The paper says the channel-wise fully connected layer of the original design is replaced by standard fully connected layers, and gives no width §3.3. The code uses a 4×4 convolution to a 4000-channel bottleneck model.py train_blending_gan.py.

The generator is trained with a combined loss eq. 2:

• L = λ · L_l2 + (1 − λ) · L_adv, with λ = 0.999 eq. 2 §3.3.
• L_l2 is the squared L2 distance between G(x) and the target x_g eq. 3.
• L_adv is the Wasserstein-style critic term, max over D of E[D(x_g) − D(G(x))] eq. 4.

Ground truth x_g is approximated by the destination image x_dst, because two frames from the same webcam are aligned §3.3 §4.1.

Gaussian-Poisson equation (phase two). Given the high-resolution source x_src, destination x_dst, mask, and the colour estimate x̃_l, the paper seeks a high-resolution image x_h that minimises H(x_h) = P(x_h) + β · C(x_h) eq. 5:

• P is a Poisson term that matches the divergence of a guidance field v to the Laplacian of x_h eq. 6. The guidance field takes the source gradient inside the mask and the destination gradient outside it eq. 8.
• C is a colour term that matches a Gaussian-blurred x_h to x̃_l eq. 7. It keeps the low frequencies of x̃_l.
• β is the colour-preserving weight, set to 1 §3.4.

The discrete form eq. 9 is ‖u − L·X_h‖² + λ · ‖G·X_h − X̃_l‖². The paper uses the symbol β in eq. 5 and λ in eq. 9 for what appears to be the same weight. Its closed form is taken from Frankot and Chellappa §3.4.

Multi-scale algorithm. The image is processed from coarse to fine on a Laplacian pyramid alg. 1. At the coarsest level the GAN output provides the colour target. At each level the paper solves eq. 9 in closed form, and the result is upsampled to become the colour target for the next, finer level. The last result is the output.

# High-resolution blending  [alg. 1] [code: gp_gan.py#L191-L228]
# Input:  obj, bg (H, W, 3) in [0, 1]; mask (H, W) in {0, 1}; G
# Output: blended (H, W, 3) in [0, 1]

def blend(obj, bg, mask, G):
    # Pyramid levels from 64 px up to the full size  [§3.4] [alg. 1]
    levels = ceil(log2(max(H, W) / 64))
    obj_pyramid = build_pyramid(obj, levels)
    bg_pyramid = build_pyramid(bg, levels)

    # Coarsest composite, mask at 64 px  [§3.3]
    # [code: gp_gan.py#L201-L202]
    composite = obj_pyramid[0] * mask_64 + bg_pyramid[0] * (1 - mask_64)

    # Phase one: realistic low-resolution colour estimate  [§3.3] [eq. 2]
    gan_image = clip(G(composite), 0, 1)

    for level in range(levels + 1):  # coarse to fine  [alg. 1]
        # Bring the previous estimate to this level's size  [alg. 1] [§3.4]
        if level > 0:
            gan_image = resize(gan_image, size_of(level))
        # Closed-form solve of eq. 9 at this level  [§3.4] [eq. 9]
        gan_image = solve_gp_equation(obj_pyramid[level],
                                      bg_pyramid[level],
                                      mask_at(level),
                                      gan_image)
        # Keep values in range after each level  [code: gp_gan.py#L149]
        gan_image = clip(gan_image, 0, 1)

    return gan_image

# Closed-form solve of the Gaussian-Poisson equation at one level  [eq. 9]
# Input:  obj, bg (h, w, 3); mask (h, w, 1); colour (h, w, 3) in [0, 1]
# Output: solution (h, w, 3)

def solve_gp_equation(obj, bg, mask, colour, beta=1.0, sigma=0.5):
    # Gradients of both images, chosen by the mask  [eq. 8]
    # [code: gp_gan.py#L140-L143]
    guidance = gradients(obj) * mask + gradients(bg) * (1 - mask)
    # Divergence of the guidance field  [eq. 6] [code: gp_gan.py#L124-L126]
    divergence = divergence_of(guidance)
    # Transform the divergence and the colour target to the solver's basis
    # [code: gp_gan.py#L134-L135]
    spectrum_div = dct2(divergence)
    spectrum_colour = dct2(colour)
    # Eigenvalues of the Laplacian and of the 3x3 Gaussian
    # [code: gp_gan.py#L88-L112]
    laplacian_eig = laplacian_eigenvalues(h, w)
    gaussian_eig = gaussian_eigenvalues(h, w, sigma)
    # Per-frequency solve, as coded (see Paper vs code)
    # [code: gp_gan.py#L128-L136]
    solution = (spectrum_div + beta * spectrum_colour) / (
        laplacian_eig + beta * gaussian_eig)
    return idct2(solution)

Key equations, if essential: the composite eq. 1; the combined loss eq. 2; the discrete objective eq. 9.

Hyperparameters

Values stated in the paper are cited to it. Values only in the code are marked as code-only, with the path and line. Items the paper does not give are "not stated".

Results

Author-reported. Setup: Transient Attributes database; 500 images with annotated object masks §4.1 §4.3. Realism scores come from RealismCNN §4.3. The user study ran on Amazon Mechanical Turk §4.4.

PB is Poisson image editing, MPB is modified Poisson editing, and MSB is multi-splines blending §4.3. The realism scores are all negative in the paper, which the authors read as evidence that many blends are still unrealistic §4.3.

Paper vs code

Code: official repository wuhuikai/GP-GAN at commit 6d1e6e51f4932bd8ba7ea8eccf640555f5c28246 (MIT). The owner wuhuikai matches the first author, Huikai Wu, and the repository is linked from the arXiv comments field, which is the first search step. The GitHub search API returned 403 from this environment, so no broader search was made. The paper itself gives no code link.

• Reconstruction loss scale. Eq. 3 uses a squared L2 norm, which is a sum over pixels eq. 3. The code uses F.mean_squared_error, a mean over all elements updater.py. With λ = 0.999, this changes the balance against the adversarial term by about the number of elements (64 × 64 × 3 = 12,288 for the training images).
• Pixel range. Images are scaled to [−1, 1] for the GAN dataset.py gp_gan.py. The paper does not state a range.
• Critic training. The paper says nothing about the critic schedule. The code uses weight clipping to ±0.01, five critic steps per generator step, and 100 steps during the first 25 iterations and every 500th iteration train_blending_gan.py updater.py. It also has a critic learning rate of 0.0002 train_blending_gan.py.
• Critic sign. Eq. 4 writes the critic as a maximum of E[D(x_g) − D(G(x))]. The code's critic minimises errD_real − errD_fake updater.py updater.py. The generator minimises D(fake) with weight 0.001 updater.py updater.py. The code's critic is therefore the negative of eq. 4's critic, and the two parts are consistent only as written in the code. Keep the signs as coded.
• Critic batch averaging. The discriminator output is averaged over the batch model.py.
• Pair sampling. The paper says two images are drawn from the same camera §4.1. The code draws the pair once when the dataset is built, then picks a random crop each time the example is read dataset.py dataset.py. The pairs are fixed for the whole run.
• Poisson solve. Eq. 9 is a squared-norm least-squares problem. Its normal equations would divide by squared operator eigenvalues. By my reading of the code, it divides by a first-power sum: laplacian_eig + beta * gaussian_eig gp_gan.py. This is derived from the algebra and I did not run it, so check it numerically before relying on it. The eigenvalues come from a DFT of a zero-padded kernel, but the solve uses DCT-II gp_gan.py.
• Gaussian kernel. The colour-term Gaussian is a 3 × 3 window built with skimage.filters.gaussian gp_gan.py. The paper gives no kernel size.
• Divergence sign. The code's L is the negative divergence of the averaged forward gradients, not the divergence as written in eq. 6 gp_gan.py. The solver uses the matching sign convention.
• Clipping. The code clips the GAN output to [0, 1] at the start and after each level gp_gan.py gp_gan.py. The paper does not mention clipping.
• Resampling. The code resizes images with cubic interpolation and masks with nearest-neighbour gp_gan.py gp_gan.py gp_gan.py. The paper does not specify resampling.
• Unsupervised variant. The repository also has an unsupervised path that optimises a 100-dimensional latent code with L-BFGS-B for up to 1000 iterations run_gp_gan.py gp_gan.py. The paper does not describe it. The card's method and results cover only the supervised Blending GAN.
• Mask type in training. The paper's mask is a central square §4.1. The code uses the same central square at 50% of the side dataset.py.
• Notation. The paper uses β in eq. 5 and λ in eq. 9 for the colour weight. The code uses color_weight for both gp_gan.py.

Pitfalls

• Phase one is supervised, not adversarial alone. The adversarial term has weight 0.001. An implementation that drops the L2 term will not match the paper's behaviour eq. 2.
• Scale of the L2 term. Using a sum instead of a mean, or the reverse, changes the effective adversarial weight by a factor of the number of elements eq. 3 updater.py.
• Critic schedule is not in the paper. Without weight clipping and the 5:1 critic ratio the adversarial term may not train as the code does updater.py.
• Solver instability near zero. The code clamps denominators whose magnitude is below 10⁻¹² to ±10⁻¹² gp_gan.py. Small denominators are numerically sensitive, so keep this guard.
• Pyramid cost. The number of levels grows with log2 of the image size, and every level runs a full closed-form solve over the whole image gp_gan.py gp_gan.py. The paper does not report run time or memory §3.4.
• User-study numbers. The votes in tab. 2 sum to 2,979. GP-GAN gets 947 of these (31.8%). The paper says "the majority of users" prefer GP-GAN §4.4, but the vote totals do not show a majority over the other methods. The number of trials is not stated. The average column matches 500 trials (for example 527 / 500 = 1.054), but that count is inferred, not stated tab. 2.
• Algorithm caption. The user-study caption says four methods were compared tab. 2. The text describes three blended images from three algorithms §4.4. The table has four rows, including GP-GAN, which is consistent with the caption.
• Evaluation code is not in the repository. The repository has no RealismCNN scoring code (searched for "realism" with no hits). The realism results cannot be reproduced from the repository README.md.
• Figure-only architecture. The exact layer widths of the Blending GAN are given only in fig. 3 and fig. 4. The code is the reliable source.

Lineage

• Builds on: Poisson image editing (Pérez et al., 2003) and modified Poisson editing (Tanaka et al., 2012) for the gradient-domain step §3.4 §4.3. Multi-splines blending (Szeliski et al., 2011) is a baseline §4.3. Wasserstein GAN (arXiv:1701.07875) is the adversarial base §3.3 README.md. The encoder-decoder follows context encoders (Pathak et al., 2016) §3.3. The pyramid-based design is fashioned after the Laplacian pyramid (Burt, 1983) §3.4.
• Compared against: PB, MPB, MSB §4.3 tab. 1 tab. 2.
• Superseded by: not stated.