A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

In plain words

Self-driving cars have to be tested in dangerous situations, such as a child running into the road, but those situations are rare in recorded driving. So engineers create them on purpose. This survey sorts the ways people do that. Some methods copy situations recorded from real traffic and vary them at random or model their statistics. Some methods place a second agent in the scene and let it search for moves that make the car fail. Others start from traffic rules and expert knowledge and build scenes that follow them. The survey also lists the simulators and datasets used for this work. It ends with five open problems: keeping scenes realistic, making the search cheap, covering many kinds of danger, working for more than one car, and controlling what gets generated. It runs no new experiments.

TL;DR

Safety-critical scenarios are rare in real driving logs, yet autonomous-vehicle evaluation needs them. This survey organises scenario-generation methods into three families: data-driven, adversarial and knowledge-based §I fig. 2. It also reviews scenario datasets tab. I, simulators tab. II and scenario design platforms, then names five challenges. It has no experiments, so the comparisons are qualitative and come from the authors.

Problem

Most driving systems are trained and evaluated on naturalistic or hand-made scenarios. Because the collision rate of a large vehicle population is very low, dangerous cases are rare in recorded data §I. The survey frames generation as an optimisation: choose the parameters of a scenario distribution so that a risk metric g of the AV system's behaviour is as high as possible §II-A, def. 3 eq. 1. Two parts make this hard: how to represent the distribution, and how to pick the metric §II-A. The risk metric decides whether the scenarios find real failures or are only trivial or too rare to matter §II-C.

Method

The survey defines the objects first, then gives a taxonomy. The sections follow the IEEEtran numbering of the compiled paper (Roman numerals for sections, letters for subsections, numbers for sub-subsections).

Definitions §II-A.

1. A driving scenario x has three parts: the static environment S (road shape, signs, lights), the initial state I of the dynamic objects, and their sequential behaviour B §II-A, def. 1.
2. The AV system F is a composition of three modules: perception takes sensor data and outputs object locations, planning produces a feasible trajectory from predicted obstacles, and control follows the waypoints §II-A, def. 2.
3. Safety-critical generation maximises the expected value of g(F(x)) over x drawn from p_theta(x) eq. 1. The survey names two difficulties: the representation of p_theta, and the choice of g §II-A.

Representation §II-B. Perception testing needs high-dimensional sensor data. Such data can be rendered with differentiable renderers and LiDAR simulators. Planning and control testing can use lower-dimensional representations: trajectories, which are more controllable but less flexible, or policy models of the other agents, which are more flexible §II-B.

Risk metrics §II-C. The survey lists these families:

• Time-based: time-to-collision for a following pair eq. 2, with variants such as time exposed TTC and modified TTC.
• Distance-based: proportion of stopping distance.
• Deceleration-based: deceleration rate to avoid the crash.
• Post-hoc unsafe-behaviour counts: collision rate, average distance driven off the road, and running stop signs or red lights.

The survey does not state a threshold at which a TTC value counts as unsafe. It also notes that the uncertainty of a risk estimate comes from modelling error and from sampling, and that sampling error falls only with more or longer simulation runs §II-C.

# Time-to-collision for a car-following pair  [eq. 2] [§II-C]
# Units of positions and speeds are not stated in the survey
def time_to_collision(x_follow, v_follow, x_lead, v_lead, lead_length):
    # Defined only when the follower is faster than the leader  [eq. 2]
    if v_follow <= v_lead:
        return None
    # Gap from the leader's rear to the follower, over the closing speed  [eq. 2]
    gap = x_lead - x_follow - lead_length
    return gap / (v_follow - v_lead)

Data-driven generation §III. The data-driven family uses only collected data. It either replays recorded situations or fits a density model and samples from it. Fitting maximises the log-likelihood of the data eq. 3.

• Direct sampling §III-A:
  • Replay: companies keep scenario libraries of important cases; tools convert logs to simulations; selection uses criteria such as cooperative actions or a non-homogeneous Poisson risk model §III-A1.
  • Clustering: unsupervised clustering groups similar scenarios; traffic primitives extracted with an HDP-HMM are proposed as building blocks §III-A2.
  • Random perturbation: sampled vehicle and pedestrian models, LiDAR models that mimic sensors, and perturbed crash parameters widen the variety §III-A3.
• Density estimation §III-B:
  • Bayesian networks and factor graphs for multi-agent structure, with importance sampling for risky cases §III-B1.
  • Deep models: LSTMs and CNNs that generate agent trajectories or surrounding agents, and a GRU-plus-CNN multi-agent model §III-B2.
  • Deep generative models: GANs, VAEs, autoregressive models, flows and diffusion models. Examples include a VAE latent space of encounter trajectories and an interpolation method that mixes normal and collision trajectories §III-B3.
  • Imitation learning, which reproduces logged behaviour and can behave unpredictably off the data §III-B4.

# Density estimation over recorded scenarios  [eq. 3] [§III-B]
# Input: logged scenarios D; model p_theta
# Output: fitted parameters theta_hat
def fit_density(scenarios, model):
    # Maximise the sum of log-likelihoods over the logged scenarios  [eq. 3]
    # The survey does not name an optimiser
    return model.maximise_log_likelihood(scenarios)

Adversarial generation §IV. A generator sets the scenario, and the AV acts as the victim. The objective minimises Q(x, pi), where Q measures the policy's performance eq. 4. The survey adds that a diversity term, such as an entropy of the distribution, is needed because the search concentrates on a small set of cases §IV. It also calls this vehicle-in-the-loop testing.

• Static scenarios §IV-A: poses of vehicles rendered into images or point clouds, attacks on detectors and segmenters, domain randomisation extended to adversarial use, and 3D-printed objects that hide a vehicle from LiDAR.
• Dynamic scenarios §IV-B:
  • Initial conditions §IV-B1: the start state or the whole trajectory is set once. Gaussian and normalising-flow models of cyclist starts, Bayesian optimisation of trajectories in point clouds, VAE-latent search for realistic traffic, and weather parameters searched with REINFORCE or cross-entropy.
  • Adversarial policy §IV-B2: a policy controls the other agents step by step, framed as reinforcement learning. Methods named include DQN, A2C, DDPG, multi-agent DDPG, constraint optimisation, Bayesian optimisation and importance-sampling variants. Adaptive stress testing (AST) uses MCTS, an LSTM for initial conditions, Go-Explore, and PPO.

# Adversarial scenario search as a loop  [§IV] [eq. 4]
# Generator: the attacker; the AV policy pi: the victim
def adversarial_generation(generator, av_policy, simulator, rounds):
    for round_index in range(rounds):
        # Sample a scenario from the generator  [§IV]
        scenario = generator.sample()
        # Run the AV in the scenario; Q measures its performance  [§IV] [eq. 4]
        q_value = simulator.evaluate(av_policy, scenario)
        # The generator is updated to lower Q  [eq. 4]
        generator.update(objective=q_value)
    return generator

Knowledge-based generation §V. Scenes must obey traffic rules and physics, so the survey adds domain knowledge K in two ways: as a condition of the distribution eq. 5, or as a constraint on existing scenarios eq. 6.

• Pre-defined rules §V-A: expert-written scenarios with parameters, natural- language or ontology descriptions, formal specifications for test cases, and whole platforms with scripted scenarios (SMARTS, MetaDrive, CARLA challenges, SUMMIT, CausalCity) §V-A.
• Knowledge-guided learning §V-B:
  • Knowledge as condition §V-B1: first-order-logic rules encoded in a tree and searched in a VAE latent space; causal graphs fed to an autoregressive model.
  • Constraint optimisation §V-B2: minimising the drivable area by evolutionary or constrained optimisation; road topology from OpenStreetMap; adversarial trajectories for trajectory predictors under feasibility constraints.

Tools §VI. The survey reviews datasets tab. I, traffic simulators tab. II, fig. 4, and scenario design platforms §VI-C. Its criteria for datasets are real versus synthetic data, bird's-eye versus first-person view, sensors, annotations, and conditions (weather, time, region, density). For simulators it looks at openness, perception realism, customisation, back-end engine, map sources and language APIs §VI-B.

Challenges §VII. Five challenges are named, with proposed directions:

• Fidelity: add traffic-rule constraints, use real data as a prior, and use neural rendering for sensors §VII-A fig. 5.
• Efficiency: surrogate models and gradient estimates for black-box AVs, and symbolic reasoning or causal discovery to guide the search §VII-B.
• Diversity: sampling-based optimisation, or regularisation and multi-modal density models §VII-C.
• Transferability: attack at a group or semantic level, and learn a hierarchical scenario that holds across different AVs §VII-D.
• Controllability: conditional generative models, with better zero-shot generalisation §VII-E. The survey adds that indoor and manipulation settings are harder to extend to §VII-F.

Guidance §VIII-B. The survey suggests three pairings: data-driven plus adversarial for broad coverage; data-driven plus knowledge-based for specific requirements that can become constraints; and adversarial plus knowledge-based when rules exist and parameters should be learned §VIII-B. Its future directions are: encode inductive bias through structure and rules, use offline RL or imitation learning to inject real data distributions, use generated scenarios for robust training with care about difficulty, and use curricula for generalisation §VIII-C.

Hyperparameters

Not applicable. The survey is a review and trains and evaluates no model §I, so no learning rates, batch sizes, model sizes or search budgets are given. Thresholds for the risk metrics are not stated §II-C.

Results

No experimental results. The survey's own comparisons are qualitative: the three generation families are compared against the five challenges in fig. 5, and the authors state that most existing methods cannot satisfy the fidelity and safety-criticality requirements at the same time fig. 1. Figures the survey itself reports, with its source:

Dataset and simulator comparisons are in the tables below, as the survey gives them.

Scenario datasets tab. I (condensed; ✓ = has it, – = not). Real, View and Time columns as the survey lists them. Annotations (2D, 3D, lane) are in tab. I itself.

The survey marks SYNTHIA and P.F.B. as not real (synthetic) tab. I.

Traffic simulators tab. II (condensed; ✓ = yes, – = no). Back-end and map columns as listed; realistic-perception and API columns are in tab. II.

Paper vs code

No official code. The survey is a review and implements no method. The LaTeX source has two GitHub links, to esmini and to OSC-ALKS-scenarios, but both are inside commented-out lines TITS_revised.tex, and both are simulator or scenario repositories, not code of this survey. No discrepancies to check.

Pitfalls

• Survey, not method. No algorithm here can be implemented as published. An agent that needs a working generator should go to the cited works, not to this page.
• Qualitative claims. The comparisons in fig. 1, fig. 2 and fig. 5 and the challenge list express the authors' judgement. No benchmark backs them.
• Symbol and sign conventions differ. Eq. 1 maximises a risk metric g. Eq. 4 minimises Q, which the text calls a performance function. Both can describe the same search, but the sign flips. Eq. 5 and eq. 6 both use K: as the knowledge in eq. 5 and as a constraint function in eq. 6 §V.
• Eq. 2 is only defined when the follower is faster than the leader. The survey writes this as a condition and does not say what to return otherwise. The units of position and speed are not stated.
• Risk-metric thresholds are absent. No TTC, PSD or DRAC value is given as "dangerous". An implementer must pick one.
• Imitation learning is described loosely. The survey calls training "exactly the same as supervised learning" §III-B4. It gives no loss or architecture.
• Sampling error is not quantified. The survey says more runs reduce it §II-C, but it gives no number of runs.
• Literature date. The text gives no literature cut-off. The arXiv record is v6, updated 2023-06-20. Later work is not covered.
• Table and figure counts. Table I is the dataset table and Table II is the simulator table in order of appearance. Figures are numbered in order, and fig. 4 is screenshots only.
• Image content. Figures are screenshots, renderings and taxonomy diagrams. Not reproduced here, per the licensing rules.

Lineage

• Builds on: other scenario-based testing surveys that the survey cites: Riedmaier et al. (scenario collection and selection), Zhong et al. (five layers of scenario-based testing) and Menzel et al. (functional, logical and concrete scenario layers). None has a card in this wiki.
• Compared against: the same three surveys. The survey says its focus is on the algorithms of safety-critical generation rather than on the whole evaluation framework §I.
• Superseded by: not known from content in this wiki.