wikillm

Paper · 2026-11-02 · cs.CV, cs.LG

Latent Diffusion-GAN: Adversarial Learning in the Autoencoded Latent Space

LD-GAN adds adversarial training to latent diffusion without a new discriminator. The frozen VAE encoder and a fine-tuned copy of its decoder form an energy on noisy latents, plus a structural term that matches encoder and decoder features. On SD-2.1 text-to-image, FID falls from 13.79 to 9.42 (author-reported). Inference uses the original decoder.

Diffusion modelsLatent diffusionGenerative adversarial network (GAN)Energy-based modelslatent-diffusionadversarial-trainingenergy-based-modelimage-generation

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Diffusion models make images by starting from noise and cleaning it up step by step. They are stable to train, but their pictures can come out a little soft, with fine textures and sharp edges blurred. GANs are better at sharpness, but they usually need a second network that judges the images, which adds cost and can make training unstable.

This paper adds the judging step without adding a second network. The image model already contains an autoencoder, which squeezes an image into a small code and rebuilds the image from it. The authors reuse that autoencoder as the judge. A code that the autoencoder can rebuild well counts as realistic. The judge also checks that the encoder and decoder agree on the same structure at several scales, which keeps the picture's layout intact. The main generator keeps its usual training, and the extra judging step is what pushes it toward sharper output.

TL;DR

LD-GAN trains a latent diffusion model with an extra adversarial term, where the discriminator is the pretrained VAE's decoder and encoder used as an energy function. The VAE encoder is frozen; a copy of the decoder is fine-tuned to act as the judge, and the original decoder is kept for generation. A structural consistency term compares encoder and decoder features at three layer pairs. Author-reported, on Stable Diffusion 2.1, FID falls from 13.79 to 9.42 and CLIP score rises from 0.3098 to 0.3428 on the paper's 20K-pair text-to-image evaluation tab. 1. The method costs about 9.6% more compute per training step and about 11.8% more memory in the authors' setup app. B.

Problem

Likelihood-based denoising training (eq. 5) can oversmooth high-frequency detail, and adversarial training in latent diffusion is hard. A discriminator in pixel space conflicts with the diffusion objective, and a separate discriminator adds parameters and instability §1 §2.2. The paper wants adversarial sharpening that keeps the structured latent prior of the pretrained VAE and does not modify the diffusion pipeline §1 §3.2.

Method

Overview. The generator is the diffusion UNet. Real latents z_t come from the VAE encoder, noised at a random timestep. Generated latents ẑ_t come from the generator. Both are scored by an energy-based discriminator built from the VAE's decoder and encoder §3.2 fig. 1.

  1. Discriminator energy. For a latent z, the energy is the squared difference between z and the encoder applied to the decoder's output Dec(z), normalised by the latent dimension d eq. 6 §3.2. The paper writes this energy in several forms (see Pitfalls).

  2. Structural consistency energy (SCE). Feature maps from matched encoder and decoder layers are compared: encoder features of z against decoder features of Dec(z) eq. 16 §3.3. Each matched pair sits at one resolution level: the encoder layer just before a downsampling step and the decoder layer just after the matching upsampling step §4.3.2. The paper uses three pairs, |L| = 3 §4.1 app. A.2 tab. 8.

  3. Total discriminator energy. D(z) = D_MSE(z) + λ_SC · D_SC(z) eq. 10, with λ_SC = 0.05 §4.1 app. A.3 tab. 9.

  4. Discriminator loss. Real latents z_t should have low energy and generated latents ẑ_t should be pushed above a margin m, in the style of energy-based GANs eq. 7 §3.2. Timesteps are sampled per example from t ~ U(0, T) §3.2. The margin m is set adaptively §4.1 (see Hyperparameters).

  5. Generator loss. The UNet is trained with its usual denoising loss plus λ_adv times the mean energy of the generated latent at a random timestep eq. 8 §3.2. λ_adv = 0.01 for SD-2.1 §4.1 app. A.1 tab. 7.

  6. Training split. The VAE encoder is frozen. Only the decoder copy is updated, so the pretrained latent space is preserved §3.2.

  7. Stabilisers. Instance normalisation replaces group normalisation in the fine-tuned decoder §3.2 §4.3.1 tab. 5. Timesteps are resampled across the whole range so the discriminator sees all noise levels §3.2.

  8. Inference. The original VAE decoder is used for image generation. The fine-tuned decoder is only used as the discriminator §3.2.

Steps as pseudocode. The paper has no algorithm box, so the step is reconstructed from [eqs. 3, 5, 7, 8] and the notes above.

Pseudocode
# LD-GAN losses for one training step  (reconstructed; no alg. box)
# Real and generated latents: (B, C, h, w)
def ld_gan_losses(images, text_embedding):
    # Encode images with the frozen VAE encoder  [§3.2]
    z_0 = encoder(images)
    # Draw one timestep per example, t ~ U(0, T)  [§3.2] [eq. 7]
    t = sample_timesteps(batch_size, T)
    eps = normal_like(z_0)
    # Real latent at timestep t from forward noising  [eq. 3]
    z_t = noise_latent(z_0, eps, t)
    # Generated latent from the generator  [§1] (form not defined in §3.2)
    z_hat = unet(z_t + eps, t, text_embedding)
    # Denoising loss  [eq. 5]
    loss_denoise = mse(unet_noise_pred(z_t, t, text_embedding), eps)
    # Energy-based discriminator loss, margin m  [eq. 7]
    loss_disc = mean(energy(z_t) + relu(margin - energy(z_hat)))
    # Generator loss: denoising plus weighted adversarial energy  [eq. 8]
    loss_gen = loss_denoise + lambda_adv * mean(energy(z_hat))
    return loss_disc, loss_gen
Pseudocode
# Discriminator energy of a latent  [eqs. 6, 10, 14, 16]
# lambda_sc = 0.05, |L| = 3 layer pairs  [§4.1]
def energy(latent):
    # Fine-tuned decoder copy, trainable  [§3.2]
    decoded = decoder_copy(latent)
    # Frozen encoder re-encodes the decoded latent  [§3.2]
    recon = encoder(decoded)
    # Pixel-like reconstruction term  [eq. 14]
    d_mse = squared_norm(latent - recon)
    # Structural term over matched layer pairs  [eq. 16] [§3.3]
    d_sc = 0
    for enc_layer, dec_layer in layer_pairs:
        feat_enc = encoder_features(latent, enc_layer)
        feat_dec = decoder_features(decoded, dec_layer)
        d_sc = d_sc + squared_norm(feat_enc - feat_dec)
    return d_mse + lambda_sc * d_sc

The energy above uses the squared-difference form of the SCE (eqs. 15–16, used in eq. 19). The paper's definition in eq. 9 is a normalised cosine term instead (see Pitfalls). Instance normalisation inside the decoder copy is not shown; it is described in §3.2.

Structural term definition. Eq. 9 is written as a sum over layers of one minus a normalised dot product, averaged by 1/|L| eq. 9. Eqs. 15, 16 and 19 use a sum of squared differences of the same features eq. 15 eq. 16 eq. 19. Both are used in the paper; see Pitfalls.

Theory. The paper gives an informal argument that the generator objective has a stationary-point convergence property under stated assumptions, and that the energy is non-negative §3.4 eq. 18. The argument relies on assumptions it calls mild (local convexity, Lipschitz behaviour) and is not a verified proof.

Hyperparameters

NameValueWhereNotes
base modelStable Diffusion 2.1 (UNet ~865M, VAE decoder ~83M, encoder ~80M)§4.1 app. Bfine-tuned from the base architecture
adversarial weight λ_adv0.01eq. 8 §4.1 app. A.1 tab. 70.05 and 0.005 both worse
SCE weight λ_SC0.05eq. 10 §4.1 app. A.3 tab. 90.1 and 0.01 both worse
feature layer pairs |L|3eq. 9 §4.1 §4.3.2 tab. 8encoder layer before each downsampling, decoder layer after the matching upsampling
margin madaptiveeq. 7 §4.1squared mean reconstruction error on the training set, recomputed every 5 epochs; ±10% changed nothing noticeable §4.1
timestep samplingt ~ U(0, T)§3.2 eq. 7T not stated; per-example or per-batch sampling not stated
decoder normalisationinstance norm replaces group norm§3.2 §4.3.1 tab. 5
trainable partsdecoder copy trainable; encoder frozen§3.2generation uses the original decoder
text-to-image training (SD-2.1)2M steps, batch 1024, lr 1e-4, 17 days§4.2.18 NVIDIA H100 GPUs §4.1; mixed precision FP16/FP32 app. B
SD-3.5-M adaptation2M steps, batch 1024, 22 days; λ_adv 0.005; λ_SC 0.05§4.2.116-channel latent; VAE scaling factor 1.5305 applied; lr stated as both 5e-5 and 1e-4 (see Pitfalls)
GLIGEN experiment400k steps, batch 64, lr 5e-5, 18 hours§4.2.2trained on Object365 and GoldG
Textual Inversion experiment5k steps, batch 4, lr 5e-3, 2.1 hours§4.2.2GSO and a custom multi-view set, 7 images per concept fig. 4
SyncDreamer experiment80k steps, batch 128, lr 5e-4 to 1e-5, 39 hours§4.2.332 Objaverse training views per object
Era3D experiment40k steps, batch 128, lr 1e-4 to 5e-5, 53 hours§4.2.3evaluation on GSO and OmniObject3D
evaluation set (text-to-image)20K image-text pairs from COCO2014 val and Flickr30k§4.2.1FID and CLIP score
ablation training datafixed subset of LAION-5B, size not statedapp. A
sampling for 2D-to-3D16 orthographic views plus 3 perspective views per azimuth§4.2.3Era3D details

Results

Author-reported. Text-to-image setup: SD-2.1 base, trained on LAION-5B, evaluated on 20K pairs from COCO2014 val and Flickr30k §4.2.1.

SettingMetricBaselineThis paperSource
SD-2.1 text-to-imageFID ↓13.799.42tab. 1
SD-2.1 text-to-imageCLIP score ↑0.30980.3428tab. 1
SD-3.5-M text-to-imageFID ↓7.9836.732tab. 2
SD-3.5-M text-to-imageCLIP score ↑0.35210.3705tab. 2
GLIGEN (bounding-box conditioning)FID ↓11.979.03tab. 3
GLIGEN (bounding-box conditioning)CLIP score ↑0.2860.318tab. 3
Textual InversionFID ↓12.658.43tab. 3
Textual InversionCLIP similarity ↑0.7540.792tab. 3
SyncDreamer, novel viewPSNR ↑ / SSIM ↑20.17 / 0.75622.32 / 0.846tab. 4
SyncDreamer, novel viewLPIPS ↓0.1360.115tab. 4
SyncDreamer, 3DCD ↓ / IoU ↑0.0232 / 0.53540.0154 / 0.6749tab. 4
Era3D, novel viewPSNR ↑ / SSIM ↑21.05 / 0.78923.83 / 0.837tab. 4
Era3D, novel viewLPIPS ↓0.1380.116tab. 4
Era3D, 3DCD ↓ / IoU ↑0.0174 / 0.65130.0125 / 0.7231tab. 4

Ablations (SD-2.1, text-to-image, same protocol as tab. 1):

AblationSettingFID ↓CLIP score ↑Source
decoder normalisationgroup norm11.020.3183tab. 5
decoder normalisationinstance norm (proposed)9.420.3428tab. 5
energy termMSE only12.110.3102tab. 6
energy termMSE + LPIPS10.740.3213tab. 6
energy termMSE + SCE (proposed)9.420.3428tab. 6
λ_adv0.05 / 0.01 / 0.00510.97 / 9.42 / 12.310.3212 / 0.3428 / 0.3187tab. 7
λ_SC0.1 / 0.05 / 0.0110.12 / 9.42 / 9.870.3246 / 0.3428 / 0.3384tab. 9
feature pairs |L|0 / 1 / 2 / 312.11 / best single 10.86 / best pair 9.87 / {1,2,3} 9.420.3102 / – / – / 0.3428tab. 8

Figure-based results are described in words only: qualitative text-to-image comparisons fig. 2 fig. 3 fig. 4 fig. 5 fig. 6 fig. 7; training curves showing FID and training loss against steps, with Diffusion-GAN diverging after about 1.3M steps fig. 8 app. B; and negative-prompt comparisons fig. 9 app. C.

Training efficiency, as the authors report it: LD-GAN reaches FID 9.42 at about 1.2M steps, where the non-adversarial baseline reaches 13.79 after 2M steps, a reduction of about 40% in steps and 34% in training time app. B fig. 8. Memory rises from about 9.7 GiB to about 10.8 GiB in mixed precision (+11.8%) and compute per training step by about 9.6% app. B.

Paper vs code

No official code. Checks: the paper text has no code link; the arXiv abstract page has no comments field and no code link; a GitHub repository search for "LD-GAN", "latent diffusion-gan", "ld-gan diffusion", "diffusion-gan autoencoded latent" and "kang jiwoo diffusion gan" returned no repository tied to the authors; the author profile Jiwoo-Kang lists only a TensorFlow fork. Three name matches (romanjacome99/LD_GAN, hdspgroup/LD-GAN, oliblum90/ld_gan) have no link to the paper and were not used. A web search was not available in this run.

Pitfalls

  • Energy definition is inconsistent in the paper. Eq. 6 divides the squared reconstruction error by the latent dimension d. Eq. 14, used in the discriminator energy of eq. 10, does not eq. 6 eq. 14. The 1/d factor changes the scale of λ_SC and the margin m. Choose one and document it.
  • Structural term has two forms. Eq. 9 uses one minus a normalised dot product of features, averaged over |L|. Eqs. 15, 16 and 19 use a sum of squared differences of features. Eq. 10 refers to D_SC from eq. 9, but the theory section relies on the squared form for non-negativity §3.4. The two forms behave differently. The paper does not say which one the experiments use.
  • Generated latent is not fully specified. Section 1 writes ẑ_t = UNet(z_t + ε) §1. Section 3.2 does not define ẑ_t, and eq. 5 uses ε_θ(z_t, t) for the same UNet. Whether ẑ_t is the predicted clean latent, the predicted noise converted to a latent, or a multi-step partial denoise is not stated. The paper also does not say whether the discriminator loss stops gradients into the generator.
  • "No extra parameters" is qualified. The abstract and §3.2 say the method adds no extra parameters. Appendix B says the method adds decoder weights, optimiser states and activations, because a copy of the decoder is fine-tuned app. B. Plan for the extra decoder copy, its optimiser state, and the memory and compute overhead.
  • SD-3.5 learning rate is contradictory. Section 4.2.1 says the learning rate was reduced to 5 × 10⁻⁵ for SD-3.5, then says both SD-3.5-M and LD-GAN were trained at 1 × 10⁻⁴ §4.2.1. The paper does not say which was used for the reported result tab. 2.
  • Margin is computed from data and an interval. m is the squared mean reconstruction error on the training set, recomputed periodically "every 5 epochs in our implementation" §4.1. The number of samples for that estimate and the implication for a 2M-step run are not stated. Results were insensitive to ±10% §4.1.
  • Timestep sampling details missing. T is not given. The paper does not say whether each example in a batch gets its own timestep or the batch shares one.
  • Ablations are single runs on an unspecified subset. The ablation subset of LAION-5B has no stated size app. A. No seeds or variances are reported. The differences in tables 5, 6, 7 and 9 are 0.1 to 2.5 FID points, so treat small gaps with caution.
  • FID sample size matters. FID uses 20K pairs §4.2.1. FID values are only comparable under the same sample count and preprocessing, which the paper keeps fixed within its tables.
  • Convergence claims are informal. Section 3.4 uses assumptions it calls mild ("local convexity", "locally Lipschitz") and states the conclusion as a local stationary-point result. Do not read it as a global convergence proof.
  • Training-curve comparison uses a different setup. Figure 8 compares Diffusion-GAN, Stable Diffusion and LD-GAN trained from scratch on an ImageNet subset app. B fig. 8. The subset size and the baseline details are not stated. This differs from the fine-tuning setup used for the main results.
  • Adaptation to a new latent width. For SD-3.5, the discriminator's decoder is modified to take 16 channels, and the VAE scaling factor 1.5305 is applied §4.2.1. The λ_adv is halved to 0.005 for stability §4.2.1. These are architecture-specific changes and need re-tuning on a new backbone.
  • Two decoders at inference. The fine-tuned decoder must never be used for generation; the original VAE decoder is kept §3.2. Mixing them up would change the output.

Lineage

  • Builds on: energy-based GANs [ZML17], which define the energy-based discriminator and margin loss used here §3.1 eq. 7. Latent diffusion models [RBL*22] are the base architecture §4.1.
  • Compared against: Diffusion-GAN [WZH23] (a discriminator on the diffusion process), Stable Diffusion 2.1 [RBL22] and 3.5-M [PEL24] as baselines, GLIGEN [LLW23] and Textual Inversion [GAA22] for conditional generation, SyncDreamer [LLZ23] and Era3D [LLL24] for 2D-to-3D generation, and LPIPS [ZIE18] as a perceptual loss tab. 6.
  • Superseded by: not stated.

Source and license

Adapted from “Latent Diffusion-GAN: Adversarial Learning in the Autoencoded Latent Space” by U-Chae Jun, Jaeeun Ko, Jiwoo Kang (arXiv:2611.02625), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.