Generative AI for Autonomous Driving: Frontiers and Opportunities

In plain words

Self-driving cars need to see, predict and plan in situations they rarely meet in practice, such as an unusual road situation or bad weather. Generative AI models learn to produce new data that looks like the real thing: pictures, laser scans, future car paths, or whole video scenes. This survey asks how those models can help a self-driving system, mainly by making more training data and test scenes, and by predicting what will happen next.

The paper first explains the main model families in everyday terms. A variational autoencoder squeezes data into a short code and rebuilds it. A generative adversarial network trains a generator against a critic that tries to spot fakes. A diffusion model learns to remove noise step by step, and a large language model predicts the next token in a sequence. It then goes through driving tasks one by one: making camera images, LiDAR scans, car trajectories, 3D occupancy grids and video, and using language models to explain or decide driving actions.

The survey also looks at how these pieces are used in practice, such as building synthetic data, running digital twins of real streets, and carrying the same ideas over to robots. It ends with open problems: rare cases, safety checks, cost, rules, ethics and environmental effects. It is a map of the field rather than a recipe, and it does not test any model itself.

TL;DR

• A survey covering the generative models and their use across the autonomous-driving stack §1, §4, §5.
• Explains VAEs, GANs, diffusion models, autoregressive models and related 3D representations as background §4.
• Groups driving applications by output: image §5.1, LiDAR §5.2, trajectory §5.3, occupancy §5.4, video §5.5, 3D/4D scenes §5.6, editing §5.7, and LLM and multimodal LLM driving §5.8, §5.9.
• Runs no experiments of its own. Numbers in its comparison tables are quoted from other papers under different datasets and settings.

Problem

Autonomous driving at SAE Level 4 and 5 needs reliable behaviour in rare and unpredictable conditions, while real data collection cannot cover every such case §1. The survey frames generative models as a way to create that data, to simulate scenes, and to forecast agent behaviour, and it names generalisation, reliability, cost and system complexity as the main technical barriers §1.

Method

The paper is a survey, so there is no method to implement. Its structure is the taxonomy:

1. Related surveys. It positions itself against seven named surveys on scenario generation, world models, video generation, LLM judging, transport planning and vision-language-action models §2.
2. Datasets. Public driving datasets are grouped into real single-vehicle perception, multi-agent prediction and planning, synthetic and simulation, and language-annotated sets, with tables for each group §3. Intro-named examples are nuScenes, Waymo Open Dataset, KITTI, Argoverse and BDD100K for real data, and CARLA, AirSim, SUMO and Isaac Sim for simulation §1.
3. Generative foundations. Background on VAEs and their conditional and vector-quantised variants §4.1, GANs and conditional GANs §4.2, diffusion models with conditioning and guidance §4.3, NeRF §4.4, 3D Gaussian splatting §4.5, the SMPL body model §4.6, and autoregressive and language models §4.7.
4. Driving applications, by modality. For each modality the survey lists methods, their venue, dataset, model family and backbone, often with a code column §5.1–§5.9. Section 5.8 covers LLM-based driving systems and §5.9 covers multimodal LLM systems.
5. Applications and beyond. Synthetic data workflows, sensor-space and trajectory generation, traffic state generation, end-to-end driving, personalised driving, digital twins, and scene understanding §6; embodied robotics §7; and open problems such as data diversity, theory, trust, V2X, traffic operations, economics, environmental impact, ethics and federated learning §8.

Foundations, as the survey states them

VAE §4.1. An encoder maps input x to a distribution over a latent z, and a decoder maps z back to data. Training maximises the evidence lower bound (ELBO). Equivalently, it minimises a reconstruction term plus a KL term that pulls the latent distribution toward a standard normal prior. Sampling draws z from that prior and decodes it. The latent is reparameterised as a mean plus a scaled noise term. The conditional version adds a condition c to encoder, decoder and prior §4.1.

VQ-VAE §4.1. The encoder output is snapped to the nearest entry in a learned codebook of K vectors. The loss has a reconstruction term, a term that moves the encoder output toward its quantised value with a stop-gradient, and a codebook term weighted by β. There is no KL term §4.1.

GAN §4.2. A generator maps noise to samples, and a discriminator scores real against generated samples. The two are trained in alternating steps. The survey lists the non-saturating generator loss, least-squares GAN and Wasserstein GAN as remedies for vanishing gradients §4.2. Conditional GANs feed a condition to both networks §4.2.

Diffusion §4.3. A forward process adds Gaussian noise over T steps with a variance schedule β_t. A reverse network is trained to undo the noise. The simplified training objective is the mean squared error between the true and predicted noise, evaluated at a random timestep §4.3. Conditioning is either by cross-attention inside a U-Net, or by classifier-free guidance, which mixes the unconditional and conditional noise predictions with a guidance scale w §4.3.

Neural radiance fields and 3D Gaussian splatting §4.4, §4.5. The survey presents both as the popular complete 3D representations, in contrast to 2D image models. NeRF describes a scene by a density and colour volume that a network is queried along rays. 3DGS describes it with 3D Gaussians, each with position, orientation, covariance and colour or intensity parameters.

# Diffusion training step, simplified objective  [§4.3]
# Input:  clean samples x0 (B, C, H, W); optional condition c
# Output: scalar loss

def diffusion_training_step(x0, c):
    # Draw one random timestep per sample  [§4.3]
    t = sample_timesteps(batch_size=B, num_steps=T)
    # Draw standard Gaussian noise of the same shape  [§4.3]
    eps = sample_normal_like(x0)                # (B, C, H, W)
    # Mix signal and noise in closed form  [§4.3]
    # alpha_bar schedule: not stated in the survey
    x_t = sqrt(alpha_bar[t]) * x0 + sqrt(1 - alpha_bar[t]) * eps
    # Predict the noise, using the condition if there is one  [§4.3]
    eps_hat = denoiser(x_t, t, c)
    # Mean squared error between true and predicted noise  [§4.3]
    return mean(square(eps - eps_hat))

# Classifier-free guided sampling  [§4.3]
# Input:  condition c; guidance scale w (value not stated)
# Output: sample x0 with the shape of the model's latent

def sample_guided(c, w):
    # Start from pure Gaussian noise  [§4.3]
    x = sample_normal(shape=(C, H, W))
    for t in range(T, 0, -1):
        # Noise predictions with and without the condition  [§4.3]
        eps_cond = denoiser(x, t, c)
        eps_uncond = denoiser(x, t, null_condition)
        # Push the prediction toward the condition by scale w  [§4.3]
        eps = eps_uncond + w * (eps_cond - eps_uncond)
        # One reverse step using the learned mean and variance  [§4.3]
        x = reverse_step(x, t, eps)
    return x

Hyperparameters

The survey trains nothing, so it states no training hyperparameters.

Results

Author-reported, and quoted from the cited works. The survey runs no experiments of its own. The values below are examples from its comparison tables. Each row uses a different dataset and setting, so the rows are not comparable with each other.

The survey also tabulates LiDAR generation (Fréchet Range Image Distance), occupancy (mean IoU where available), video (FVD on nuScenes), 3D/4D scene generation, scene editing and LLM/MLLM driving systems §5.2, §5.4–§5.9. Those values are in the source tables and were not all transcribed here.

Paper vs code

No official code. The paper's abstract links a repository of cited works (taco-group/GenAI4AD), which is a paper index, not code. It was not cloned. The per-method tables list GitHub links for some methods and "N/A" for others. Those links were not checked.

Pitfalls

• Different settings in one table. The trajectory table mixes ETH/UCY, Stanford Drone, nuScenes, WOMD, rounD, highD and other datasets, and the image table mixes nuScenes with Google Street View. Do not rank methods across rows that use different datasets or metrics §5.1, §5.3.
• Quoted numbers are not verified. The values come from each original paper, under that paper's protocol. The survey does not reproduce them.
• Guidance scale and schedule are not given. Classifier-free guidance is described by formula only, and the diffusion schedule is named but not specified §4.3.
• VQ-VAE weighting differs from the common form. The survey puts β on the codebook term and leaves the encoder commitment term unweighted §4.1. This is the reverse of the placement in the original VQ-VAE paper, which was not read for this card. Check that paper before implementing this loss.
• Symbol mismatch in the GAN section. The text refers to the discriminator parameters as ψ, but the equation uses ρ, and the generator is written as D_θ, a letter often used for discriminators §4.2.
• Some rows are commented out in the source. Entries such as MagicDrive3D, HoloDrive, WeatherDG and UrbanArchitect are commented out of the image table in the source. They are not in the published tables and are not cited here.
• Tables are long. Several methods appear with "N/A" as their code link, so a reader cannot always reproduce a method from this survey alone.

Lineage

• Related surveys named in the paper: data-driven scenario generation for automated vehicle testing; safety-critical scenario generation; world models for autonomous driving; video generation and world models; LLM-as-a-judge; generative AI in transportation planning; vision-language-action models §2.
• Concept pages that cover its generative families: diffusion models, generative adversarial networks, variational autoencoders and world models.
• Superseded by: not known from content in this wiki.