Paper · 2026-11-02 · cs.CV, cs.LG
Latent Diffusion-GAN: Adversarial Learning in the Autoencoded Latent Space
LD-GAN adds adversarial training to latent diffusion without a new discriminator. The frozen VAE encoder and a fine-tuned copy of its decoder form an energy on noisy latents, plus a structural term that matches encoder and decoder features. On SD-2.1 text-to-image, FID falls from 13.79 to 9.42 (author-reported). Inference uses the original decoder.
Diffusion modelsLatent diffusionGenerative adversarial network (GAN)Energy-based modelslatent-diffusionadversarial-trainingenergy-based-modelimage-generation
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Diffusion models make images by starting from noise and cleaning it up step by step. They are stable to train, but their pictures can come out a little soft, with fine textures and sharp edges blurred. GANs are better at sharpness, but they usually need a second network that judges the images, which adds cost and can make training unstable.
This paper adds the judging step without adding a second network. The image model already contains an autoencoder, which squeezes an image into a small code and rebuilds the image from it. The authors reuse that autoencoder as the judge. A code that the autoencoder can rebuild well counts as realistic. The judge also checks that the encoder and decoder agree on the same structure at several scales, which keeps the picture's layout intact. The main generator keeps its usual training, and the extra judging step is what pushes it toward sharper output.
TL;DR
LD-GAN trains a latent diffusion model with an extra adversarial term, where the discriminator is the pretrained VAE's decoder and encoder used as an energy function. The VAE encoder is frozen; a copy of the decoder is fine-tuned to act as the judge, and the original decoder is kept for generation. A structural consistency term compares encoder and decoder features at three layer pairs. Author-reported, on Stable Diffusion 2.1, FID falls from 13.79 to 9.42 and CLIP score rises from 0.3098 to 0.3428 on the paper's 20K-pair text-to-image evaluation tab. 1. The method costs about 9.6% more compute per training step and about 11.8% more memory in the authors' setup app. B.
Problem
Likelihood-based denoising training (eq. 5) can oversmooth high-frequency detail, and adversarial training in latent diffusion is hard. A discriminator in pixel space conflicts with the diffusion objective, and a separate discriminator adds parameters and instability §1 §2.2. The paper wants adversarial sharpening that keeps the structured latent prior of the pretrained VAE and does not modify the diffusion pipeline §1 §3.2.
Method
Overview. The generator is the diffusion UNet. Real latents z_t come from
the VAE encoder, noised at a random timestep. Generated latents ẑ_t come from
the generator. Both are scored by an energy-based discriminator built from the
VAE's decoder and encoder §3.2 fig. 1.
-
Discriminator energy. For a latent
z, the energy is the squared difference betweenzand the encoder applied to the decoder's outputDec(z), normalised by the latent dimensiondeq. 6 §3.2. The paper writes this energy in several forms (see Pitfalls). -
Structural consistency energy (SCE). Feature maps from matched encoder and decoder layers are compared: encoder features of
zagainst decoder features ofDec(z)eq. 16 §3.3. Each matched pair sits at one resolution level: the encoder layer just before a downsampling step and the decoder layer just after the matching upsampling step §4.3.2. The paper uses three pairs,|L| = 3§4.1 app. A.2 tab. 8. -
Total discriminator energy.
D(z) = D_MSE(z) + λ_SC · D_SC(z)eq. 10, withλ_SC = 0.05§4.1 app. A.3 tab. 9. -
Discriminator loss. Real latents
z_tshould have low energy and generated latentsẑ_tshould be pushed above a marginm, in the style of energy-based GANs eq. 7 §3.2. Timesteps are sampled per example fromt ~ U(0, T)§3.2. The marginmis set adaptively §4.1 (see Hyperparameters). -
Generator loss. The UNet is trained with its usual denoising loss plus
λ_advtimes the mean energy of the generated latent at a random timestep eq. 8 §3.2.λ_adv = 0.01for SD-2.1 §4.1 app. A.1 tab. 7. -
Training split. The VAE encoder is frozen. Only the decoder copy is updated, so the pretrained latent space is preserved §3.2.
-
Stabilisers. Instance normalisation replaces group normalisation in the fine-tuned decoder §3.2 §4.3.1 tab. 5. Timesteps are resampled across the whole range so the discriminator sees all noise levels §3.2.
-
Inference. The original VAE decoder is used for image generation. The fine-tuned decoder is only used as the discriminator §3.2.
Steps as pseudocode. The paper has no algorithm box, so the step is reconstructed from [eqs. 3, 5, 7, 8] and the notes above.
# LD-GAN losses for one training step (reconstructed; no alg. box)
# Real and generated latents: (B, C, h, w)
def ld_gan_losses(images, text_embedding):
# Encode images with the frozen VAE encoder [§3.2]
z_0 = encoder(images)
# Draw one timestep per example, t ~ U(0, T) [§3.2] [eq. 7]
t = sample_timesteps(batch_size, T)
eps = normal_like(z_0)
# Real latent at timestep t from forward noising [eq. 3]
z_t = noise_latent(z_0, eps, t)
# Generated latent from the generator [§1] (form not defined in §3.2)
z_hat = unet(z_t + eps, t, text_embedding)
# Denoising loss [eq. 5]
loss_denoise = mse(unet_noise_pred(z_t, t, text_embedding), eps)
# Energy-based discriminator loss, margin m [eq. 7]
loss_disc = mean(energy(z_t) + relu(margin - energy(z_hat)))
# Generator loss: denoising plus weighted adversarial energy [eq. 8]
loss_gen = loss_denoise + lambda_adv * mean(energy(z_hat))
return loss_disc, loss_gen# Discriminator energy of a latent [eqs. 6, 10, 14, 16]
# lambda_sc = 0.05, |L| = 3 layer pairs [§4.1]
def energy(latent):
# Fine-tuned decoder copy, trainable [§3.2]
decoded = decoder_copy(latent)
# Frozen encoder re-encodes the decoded latent [§3.2]
recon = encoder(decoded)
# Pixel-like reconstruction term [eq. 14]
d_mse = squared_norm(latent - recon)
# Structural term over matched layer pairs [eq. 16] [§3.3]
d_sc = 0
for enc_layer, dec_layer in layer_pairs:
feat_enc = encoder_features(latent, enc_layer)
feat_dec = decoder_features(decoded, dec_layer)
d_sc = d_sc + squared_norm(feat_enc - feat_dec)
return d_mse + lambda_sc * d_scThe energy above uses the squared-difference form of the SCE (eqs. 15–16, used in eq. 19). The paper's definition in eq. 9 is a normalised cosine term instead (see Pitfalls). Instance normalisation inside the decoder copy is not shown; it is described in §3.2.
Structural term definition. Eq. 9 is written as a sum over layers of one
minus a normalised dot product, averaged by 1/|L| eq. 9. Eqs. 15, 16 and 19
use a sum of squared differences of the same features eq. 15 eq. 16
eq. 19. Both are used in the paper; see Pitfalls.
Theory. The paper gives an informal argument that the generator objective has a stationary-point convergence property under stated assumptions, and that the energy is non-negative §3.4 eq. 18. The argument relies on assumptions it calls mild (local convexity, Lipschitz behaviour) and is not a verified proof.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| base model | Stable Diffusion 2.1 (UNet ~865M, VAE decoder ~83M, encoder ~80M) | §4.1 app. B | fine-tuned from the base architecture |
adversarial weight λ_adv | 0.01 | eq. 8 §4.1 app. A.1 tab. 7 | 0.05 and 0.005 both worse |
SCE weight λ_SC | 0.05 | eq. 10 §4.1 app. A.3 tab. 9 | 0.1 and 0.01 both worse |
feature layer pairs |L| | 3 | eq. 9 §4.1 §4.3.2 tab. 8 | encoder layer before each downsampling, decoder layer after the matching upsampling |
margin m | adaptive | eq. 7 §4.1 | squared mean reconstruction error on the training set, recomputed every 5 epochs; ±10% changed nothing noticeable §4.1 |
| timestep sampling | t ~ U(0, T) | §3.2 eq. 7 | T not stated; per-example or per-batch sampling not stated |
| decoder normalisation | instance norm replaces group norm | §3.2 §4.3.1 tab. 5 | |
| trainable parts | decoder copy trainable; encoder frozen | §3.2 | generation uses the original decoder |
| text-to-image training (SD-2.1) | 2M steps, batch 1024, lr 1e-4, 17 days | §4.2.1 | 8 NVIDIA H100 GPUs §4.1; mixed precision FP16/FP32 app. B |
| SD-3.5-M adaptation | 2M steps, batch 1024, 22 days; λ_adv 0.005; λ_SC 0.05 | §4.2.1 | 16-channel latent; VAE scaling factor 1.5305 applied; lr stated as both 5e-5 and 1e-4 (see Pitfalls) |
| GLIGEN experiment | 400k steps, batch 64, lr 5e-5, 18 hours | §4.2.2 | trained on Object365 and GoldG |
| Textual Inversion experiment | 5k steps, batch 4, lr 5e-3, 2.1 hours | §4.2.2 | GSO and a custom multi-view set, 7 images per concept fig. 4 |
| SyncDreamer experiment | 80k steps, batch 128, lr 5e-4 to 1e-5, 39 hours | §4.2.3 | 32 Objaverse training views per object |
| Era3D experiment | 40k steps, batch 128, lr 1e-4 to 5e-5, 53 hours | §4.2.3 | evaluation on GSO and OmniObject3D |
| evaluation set (text-to-image) | 20K image-text pairs from COCO2014 val and Flickr30k | §4.2.1 | FID and CLIP score |
| ablation training data | fixed subset of LAION-5B, size not stated | app. A | |
| sampling for 2D-to-3D | 16 orthographic views plus 3 perspective views per azimuth | §4.2.3 | Era3D details |
Results
Author-reported. Text-to-image setup: SD-2.1 base, trained on LAION-5B, evaluated on 20K pairs from COCO2014 val and Flickr30k §4.2.1.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| SD-2.1 text-to-image | FID ↓ | 13.79 | 9.42 | tab. 1 |
| SD-2.1 text-to-image | CLIP score ↑ | 0.3098 | 0.3428 | tab. 1 |
| SD-3.5-M text-to-image | FID ↓ | 7.983 | 6.732 | tab. 2 |
| SD-3.5-M text-to-image | CLIP score ↑ | 0.3521 | 0.3705 | tab. 2 |
| GLIGEN (bounding-box conditioning) | FID ↓ | 11.97 | 9.03 | tab. 3 |
| GLIGEN (bounding-box conditioning) | CLIP score ↑ | 0.286 | 0.318 | tab. 3 |
| Textual Inversion | FID ↓ | 12.65 | 8.43 | tab. 3 |
| Textual Inversion | CLIP similarity ↑ | 0.754 | 0.792 | tab. 3 |
| SyncDreamer, novel view | PSNR ↑ / SSIM ↑ | 20.17 / 0.756 | 22.32 / 0.846 | tab. 4 |
| SyncDreamer, novel view | LPIPS ↓ | 0.136 | 0.115 | tab. 4 |
| SyncDreamer, 3D | CD ↓ / IoU ↑ | 0.0232 / 0.5354 | 0.0154 / 0.6749 | tab. 4 |
| Era3D, novel view | PSNR ↑ / SSIM ↑ | 21.05 / 0.789 | 23.83 / 0.837 | tab. 4 |
| Era3D, novel view | LPIPS ↓ | 0.138 | 0.116 | tab. 4 |
| Era3D, 3D | CD ↓ / IoU ↑ | 0.0174 / 0.6513 | 0.0125 / 0.7231 | tab. 4 |
Ablations (SD-2.1, text-to-image, same protocol as tab. 1):
| Ablation | Setting | FID ↓ | CLIP score ↑ | Source |
|---|---|---|---|---|
| decoder normalisation | group norm | 11.02 | 0.3183 | tab. 5 |
| decoder normalisation | instance norm (proposed) | 9.42 | 0.3428 | tab. 5 |
| energy term | MSE only | 12.11 | 0.3102 | tab. 6 |
| energy term | MSE + LPIPS | 10.74 | 0.3213 | tab. 6 |
| energy term | MSE + SCE (proposed) | 9.42 | 0.3428 | tab. 6 |
λ_adv | 0.05 / 0.01 / 0.005 | 10.97 / 9.42 / 12.31 | 0.3212 / 0.3428 / 0.3187 | tab. 7 |
λ_SC | 0.1 / 0.05 / 0.01 | 10.12 / 9.42 / 9.87 | 0.3246 / 0.3428 / 0.3384 | tab. 9 |
feature pairs |L| | 0 / 1 / 2 / 3 | 12.11 / best single 10.86 / best pair 9.87 / {1,2,3} 9.42 | 0.3102 / – / – / 0.3428 | tab. 8 |
Figure-based results are described in words only: qualitative text-to-image comparisons fig. 2 fig. 3 fig. 4 fig. 5 fig. 6 fig. 7; training curves showing FID and training loss against steps, with Diffusion-GAN diverging after about 1.3M steps fig. 8 app. B; and negative-prompt comparisons fig. 9 app. C.
Training efficiency, as the authors report it: LD-GAN reaches FID 9.42 at about 1.2M steps, where the non-adversarial baseline reaches 13.79 after 2M steps, a reduction of about 40% in steps and 34% in training time app. B fig. 8. Memory rises from about 9.7 GiB to about 10.8 GiB in mixed precision (+11.8%) and compute per training step by about 9.6% app. B.
Paper vs code
No official code. Checks: the paper text has no code link; the arXiv abstract
page has no comments field and no code link; a GitHub repository search for
"LD-GAN", "latent diffusion-gan", "ld-gan diffusion", "diffusion-gan autoencoded
latent" and "kang jiwoo diffusion gan" returned no repository tied to the
authors; the author profile Jiwoo-Kang lists only a TensorFlow fork.
Three name matches (romanjacome99/LD_GAN, hdspgroup/LD-GAN,
oliblum90/ld_gan) have no link to the paper and were not used. A web search
was not available in this run.
Pitfalls
- Energy definition is inconsistent in the paper. Eq. 6 divides the squared
reconstruction error by the latent dimension
d. Eq. 14, used in the discriminator energy of eq. 10, does not eq. 6 eq. 14. The1/dfactor changes the scale ofλ_SCand the marginm. Choose one and document it. - Structural term has two forms. Eq. 9 uses one minus a normalised dot
product of features, averaged over
|L|. Eqs. 15, 16 and 19 use a sum of squared differences of features. Eq. 10 refers toD_SCfrom eq. 9, but the theory section relies on the squared form for non-negativity §3.4. The two forms behave differently. The paper does not say which one the experiments use. - Generated latent is not fully specified. Section 1 writes
ẑ_t = UNet(z_t + ε)§1. Section 3.2 does not defineẑ_t, and eq. 5 usesε_θ(z_t, t)for the same UNet. Whetherẑ_tis the predicted clean latent, the predicted noise converted to a latent, or a multi-step partial denoise is not stated. The paper also does not say whether the discriminator loss stops gradients into the generator. - "No extra parameters" is qualified. The abstract and §3.2 say the method adds no extra parameters. Appendix B says the method adds decoder weights, optimiser states and activations, because a copy of the decoder is fine-tuned app. B. Plan for the extra decoder copy, its optimiser state, and the memory and compute overhead.
- SD-3.5 learning rate is contradictory. Section 4.2.1 says the learning rate
was reduced to
5 × 10⁻⁵for SD-3.5, then says both SD-3.5-M and LD-GAN were trained at1 × 10⁻⁴§4.2.1. The paper does not say which was used for the reported result tab. 2. - Margin is computed from data and an interval.
mis the squared mean reconstruction error on the training set, recomputed periodically "every 5 epochs in our implementation" §4.1. The number of samples for that estimate and the implication for a 2M-step run are not stated. Results were insensitive to ±10% §4.1. - Timestep sampling details missing.
Tis not given. The paper does not say whether each example in a batch gets its own timestep or the batch shares one. - Ablations are single runs on an unspecified subset. The ablation subset of LAION-5B has no stated size app. A. No seeds or variances are reported. The differences in tables 5, 6, 7 and 9 are 0.1 to 2.5 FID points, so treat small gaps with caution.
- FID sample size matters. FID uses 20K pairs §4.2.1. FID values are only comparable under the same sample count and preprocessing, which the paper keeps fixed within its tables.
- Convergence claims are informal. Section 3.4 uses assumptions it calls mild ("local convexity", "locally Lipschitz") and states the conclusion as a local stationary-point result. Do not read it as a global convergence proof.
- Training-curve comparison uses a different setup. Figure 8 compares Diffusion-GAN, Stable Diffusion and LD-GAN trained from scratch on an ImageNet subset app. B fig. 8. The subset size and the baseline details are not stated. This differs from the fine-tuning setup used for the main results.
- Adaptation to a new latent width. For SD-3.5, the discriminator's decoder is
modified to take 16 channels, and the VAE scaling factor 1.5305 is applied
§4.2.1. The
λ_advis halved to 0.005 for stability §4.2.1. These are architecture-specific changes and need re-tuning on a new backbone. - Two decoders at inference. The fine-tuned decoder must never be used for generation; the original VAE decoder is kept §3.2. Mixing them up would change the output.
Lineage
- Builds on: energy-based GANs [ZML17], which define the energy-based discriminator and margin loss used here §3.1 eq. 7. Latent diffusion models [RBL*22] are the base architecture §4.1.
- Compared against: Diffusion-GAN [WZH23] (a discriminator on the diffusion process), Stable Diffusion 2.1 [RBL22] and 3.5-M [PEL24] as baselines, GLIGEN [LLW23] and Textual Inversion [GAA22] for conditional generation, SyncDreamer [LLZ23] and Era3D [LLL24] for 2D-to-3D generation, and LPIPS [ZIE18] as a perceptual loss tab. 6.
- Superseded by: not stated.
Source and license
Adapted from “Latent Diffusion-GAN: Adversarial Learning in the Autoencoded Latent Space” by U-Chae Jun, Jaeeun Ko, Jiwoo Kang (arXiv:2611.02625), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.