Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers

In plain words

This paper is a long tour of generative AI: systems that make new content such as text, images, audio and video. It covers the model families behind today's tools. Autoencoders squeeze data into a short code and rebuild it from that code. Variational autoencoders learn a range of codes, so new samples can be drawn from them. Transformers learn which parts of an input matter most for each other part. GANs pit a generator against a discriminator. Diffusion models learn to remove noise step by step.

The paper then lists many generative tools and the tasks they cover, from text and images to code, music and speech. It also covers applications in media, education, healthcare and business, and discusses risks such as job shifts, cyber threats and impersonation.

It runs no experiments of its own. Read it for a map of the field and its vocabulary. For the actual method details, go to the original papers it cites.

TL;DR

A survey, not a method. It explains the main generative model families and their variants, lists tools and applications, and discusses challenges. It gives no algorithm to implement and no benchmark of its own. Use it for orientation, then read the cited originals (for example the GPT, GAN and diffusion papers) for numbers and procedures.

Problem

The paper's stated aim is an in-depth overview of the state of generative AI: models, task categories, applications, challenges and prospects §I. It does not solve a technical problem itself.

The method-level motivations it gives are these:

• Recurrent and convolutional networks struggle with long-range dependencies, gradient vanishing and explosion, and sequential training that does not parallelise §II.B.
• GANs suffer from mode collapse, non-convergence and sensitivity to hyperparameters such as learning rate and batch size §II.C.2.

The paper states no assumptions of its own about data or compute.

Method

This is a review, so there is no algorithm of its own. Its content, with the paper's own section and equation numbers:

1. Autoencoders §II.A: an encoder maps input to a code, and a decoder maps the code back to an output eq. 1 eq. 2. Training minimises a dissimilarity between input and output eq. 3. The paper lists plain, contractive, denoising and sparse variants §II.A. It describes plain autoencoders as not generative.
2. Variational autoencoders §II.A.1: the encoder outputs a distribution over a latent variable, a sample is drawn from it, and the decoder reconstructs the input. Training adds regularisation. The paper writes the joint distribution as a likelihood times a prior eq. 4 and the posterior via Bayes' rule eq. 5. It notes that approximation quality can be assessed with Kullback-Leibler divergence §II.A.1. Its application table lists image, audio, video, sensor and finance uses tab. 1.
3. Transformers §II.B: stacked self-attention and feed-forward layers, with residual connections and layer normalisation, form encoder and decoder stacks. The scaled dot-product attention is given in eq. 6 and multi-head attention in eq. 7 fig. 5 fig. 6. The paper describes the original encoder as a stack of N = 6 identical layers §II.B.
4. GPT series §II.B.1 to §II.B.4: GPT-1 is a 12-layer decoder-only transformer pretrained on unlabelled text and then fine-tuned on labelled data fig. 7. GPT-2 (2019) and GPT-3 (2020) scale this up. GPT-4 (March 2023) is described as multimodal and as outperforming the compared models on MMLU. The paper calls GPT-4 "the most recent" GPT model §II.B.4.
5. GANs §II.C.1: a generator maps noise to fake samples and a discriminator classifies real versus fake. The discriminator uses a sigmoid output and binary cross-entropy. The minimax objective is eq. 8.
6. GAN challenges and variants §II.C.2 §II.C.3: mode collapse, non-convergence and vanishing gradients. Variants covered: conditional GAN §II.C.3; DCGAN, which uses strided and fractional-strided convolutions with batch normalisation §II.C.3; Wasserstein GAN, which uses the Wasserstein distance eq. 9 and enforces the Lipschitz constraint by weight clipping §II.C.3; CycleGAN, with two generators and three losses §II.C.3; and StarGAN §II.C.3. Other named GANs: PGAN, BigGAN, StyleGAN and StyleGAN2, InfoGAN, Stacked GAN and BiGAN §II.C.3.
7. Diffusion models §II.D: a forward process adds Gaussian noise over steps, and a reverse process denoises step by step to reconstruct data. The section states no equations and no training details.
8. Tasks §III, tools tab. 2, applications §IV and outlook §V: the tool table lists 73 numbered entries with developer, task, year and description. The applications cover media and entertainment, education, healthcare and business. The outlook covers job shifts, cybersecurity and impersonation.

The formulas, as pseudocode with the paper's citations:

# Scaled dot-product attention  [eq. 6]
# Input:  queries (n, d_k), keys (m, d_k), values (m, d_v)
# Output: outputs (n, d_v)

def attention(queries, keys, values):
    # Score each query against each key, scaled by sqrt(d_k)  [eq. 6]
    scores = queries @ keys.T / sqrt(d_k)    # (n, m)
    # Normalise each row of scores into weights  [eq. 6]
    weights = softmax(scores, axis=-1)       # (n, m)
    # Weighted sum of the values  [eq. 6]
    return weights @ values                  # (n, d_v)

# Multi-head attention  [eq. 7]
# Input:  queries, keys, values; one projection set per head
# Output: (n, d_model)

def multi_head(queries, keys, values, heads, W_o):
    # Project the inputs separately for each head  [eq. 7]
    head_outputs = []
    for head in heads:
        q = queries @ head.W_q
        k = keys @ head.W_k
        v = values @ head.W_v
        head_outputs.append(attention(q, k, v))
    # Concatenate the head outputs and project them back  [eq. 7]
    return concat(head_outputs) @ W_o

# GAN minimax objective  [eq. 8]
# Discriminator maximises this value; generator minimises it
# Input:  real_batch, noise_batch
# Output: scalar value

def gan_value(discriminator, generator, real_batch, noise_batch):
    # Mean log-score the discriminator gives real samples  [eq. 8]
    real_term = mean(log(discriminator(real_batch)))
    # Mean log(1 - score) on generated samples  [eq. 8]
    fake_term = mean(log(1 - discriminator(generator(noise_batch))))
    return real_term + fake_term

The equations these blocks transcribe, in the paper's notation: eq. 6 is the attention weights; eq. 7 is the head concatenation; eq. 8 is the minimax objective, printed as min over G of max over D.

Hyperparameters

Only the GPT settings are concrete. The paper gives them as a list without a source table, and does not cite the original configuration beyond the section. Everything else in the table is "not stated".

Results

Author-reported. This paper runs no benchmark. Its comparisons are visual or qualitative, and their setup is listed here.

Setup: model versions, sampling settings, number of runs and hardware are not stated for any comparison. Screenshots are the only evidence.

Paper vs code

No official code. The paper has no repository link. Its only footnote points to a blog article, on the autoencoder figure fig. 1. A GitHub repository search for the arXiv id and for title words returned no results. A survey has no implementation to compare against.

Pitfalls

• No method to implement. Nothing here is a reproducible procedure beyond the formulas above. Implementers should take procedures from the original papers, such as GPT §II.B, GAN §II.C and diffusion §II.D, and check them against those papers.
• Dated. The paper reflects the state of the field in late 2023. It calls GPT-4 "the most recent" GPT model §II.B.4. The tool list tab. 2, the market figures and the "current" language are snapshots from that time and were not updated for this card.
• Attribution of VAEs. The text credits Kingma et al. [35] with introducing variational inference into autoencoders §II.A.1. In the reference list, [35] is a 2014 semi-supervised learning paper, and [36] is the VAE tutorial. Check the original before citing this claim.
• Equation 8 and its prose. The equation is printed as min over G of max over D. The prose in §II.C.1 describes each player's goal with "maximize" and "minimize" in a way that is easy to misread. Use the equation.
• Notation drift. Equations 1 to 3 use X_i in one place and X_t in another, and eq. 3 names no dissimilarity measure or norm. The paper states no loss function for autoencoders.
• Two transformer configurations. The original encoder stack (N = 6) §II.B and GPT-1 (12 layers, decoder-only) §II.B.1 are different models. Do not mix their settings.
• Table 3 not found. Section IV.A refers to "Table 3" §IV.A. It was not found in the pages read.
• Uncited or third-party numbers. The market forecast ($137 billion in 2023, $1.3 trillion by 2030) §IV.D, the 84% and 47% workforce figures §V.B and the job-task lists are cited to third parties. They were not verified here.
• Table 2 data quality. Several year fields are missing or "N/A" (rows 41, 43, 46, 47, 71, 72), and row 66 has no description. Treat it as a list of names, not a data source.
• Figures. Several figures are screenshots of third-party products or are taken from web articles. This card reproduces none of them and cites them only by number.

Lineage

• Builds on: the paper's cited originals, none with a card in this wiki yet. Transformers, Vaswani et al. §II.B [24]. GANs, Goodfellow et al. §II.C.1 [25]. VAEs, Kingma et al. §II.A.1 [35] and [36]. GPT, Radford et al. §II.B.1 [65].
• Compared against: ChatGPT vs Bard, and Firefly vs Stable Diffusion, qualitatively fig. 9 fig. 10. No quantitative baselines.
• Superseded by: not known from wiki content.