---
kind: concept
slug: fuzz-testing
title: Fuzz testing
summary: >-
  Fuzz testing searches for inputs that make a system fail by mutating candidates
  and keeping those that expose new failures. This page covers simulation-based
  fuzzing of driving software, where a candidate is a traffic scenario. One card
  so far: a genetic fuzzer with a learned violation score, author-reported to beat
  three baselines on one system and one map.
tags: [testing, simulation-testing, autonomous-driving, genetic-algorithm]
papers: ["2412.13802"]
related: [metaheuristic-optimization, surrogate-modeling, self-attention, adversarial-evaluation, functional-safety]
status: draft
updated_at: 2026-10-25
---

## Overview

Fuzz testing is a search over test inputs. The system under test is run on a
candidate input, the run is scored for signs of failure, and promising inputs
are kept and varied. For self-driving software the input is a whole scenario:
routes for the ego vehicle and other traffic, pedestrians and weather
[arxiv:2412.13802 §4.2]. The
problem is that the space of scenarios is very large, so the question is which
ones to run next [arxiv:2412.13802 §1].

The shared core is a population of scenarios that is scored by feedback,
selected, crossed over and mutated [arxiv:2412.13802 §4.2]. What differs
between designs is the feedback, the score and how the variation is done. The
only card in this wiki so far covers one such design; see Variants.

## Variants

### Simulation-feedback genetic fuzzing (SimADFuzz)

The fuzzer runs each scenario in CARLA and reads per-vehicle position, speed,
acceleration and heading at every frame [arxiv:2412.13802 §4.1] [tab. 2]. A
Transformer encoder reads that sequence and outputs a violation score that is
one of four fitness objectives. The others are a road-feature classifier
(SDC-Scissor), the minimum distance to the ego vehicle and the count of unique
violations. These are combined with NSGA-II [arxiv:2412.13802 §4.2.1]. The
design changes two parts of earlier genetic fuzzers. The score reads the whole
run instead of one aggregate, and mutation removes vehicles that are stuck or
moving away from the ego vehicle instead of changing them at random
[arxiv:2412.13802 §3] [alg. 1]. Card: [SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems](/papers/2412.13802).

Author-reported results use InterFuser on Town03 with 6-hour runs. It finds 35
unique violations, against 3 for AV-Fuzzer, 18 for DriveFuzz and 26 for
TM-Fuzzer [arxiv:2412.13802 §5.2.2] [fig. 8]. Its first collision comes at 17
minutes, against 21 minutes for TM-Fuzzer [arxiv:2412.13802 §5.2.2].

## When to use which

The choice depends on what you can run and what you can build. The only design
with evidence here is the simulation-feedback one, so the rows below are about
its constraints.

| Situation | Use | Why |
|---|---|---|
| You have a driving simulator that logs per-vehicle state each frame and a GPU | Simulation-feedback genetic fuzzing | The method is built on that feedback [arxiv:2412.13802 §4.1]. The reference runs used CARLA at 20 Hz with 10-minute simulations [arxiv:2412.13802 §5.1.2]. |
| You can afford hours of simulation per campaign and want many distinct failures | Simulation-feedback genetic fuzzing | The comparison runs were 6 hours per fuzzer [arxiv:2412.13802 §5.2.2]. The gain over baselines is author-reported. |
| You cannot produce labelled training scenarios for a learned score | Not covered by this page | The learned score is trained on 1,000 scenarios of 2 minutes each [arxiv:2412.13802 §5.1.2]. No cited result shows the method without it. |
| You need results for a system other than InterFuser or on another map | Re-run before relying on the numbers | Only one system and one map were evaluated, and the authors list this as a threat to validity [arxiv:2412.13802 §5.3]. |
| You need the gain attributed to one component | Not answerable from this page | The 3-hour ablation compares a random baseline with a combined setting [arxiv:2412.13802 §5.2.1]. The cited results do not isolate the learned score. |

## Implementation sketch

Language-agnostic outline of the loop in the simulation-feedback design. Where
the variants differ, the choices are marked. Parameters are the paper's
[arxiv:2412.13802 §5.1.2].

```pseudocode
# Seed a population of random scenarios  [arxiv:2412.13802 §5.1.2]
population = [random_scenario() for _ in range(20)]

# Repeat until the time budget is spent  [arxiv:2412.13802 §5.2]
while not budget_spent():
    # Run each scenario and record per-vehicle states per frame  [§4.1]
    traces = [simulate(s) for s in population]

    # Score each scenario on several objectives, e.g. a learned score,
    # a road score, minimum distance and unique violations  [§4.2.1]
    scores = [objectives(s, t) for s, t in zip(population, traces)]

    # Make offspring; each operator runs with probability 0.5  [§5.1.2]
    children = []
    for pair in pick_parents(population, scores):
        # Swap routes between parents, keeping the ego route  [§4.2.2]
        child = crossover(pair)
        # Remove stuck or leaving NPCs and redraw the rest  [§4.2.3]
        child = mutate(child)
        children.append(child)

    # Keep the best of parents and children under NSGA-II  [§4.2.1]
    population = nsga2_select(population + children, scores)

# Report violations found in the run  [§4.3]
return unique_violations(population)
```

Variants differ in the objectives used (this design uses four
[arxiv:2412.13802 §4.2.1]), in the mutation rule [arxiv:2412.13802 §4.2.3], and
in the run length [arxiv:2412.13802 §5.2].

## Contradictions

Within the one card so far: the 6-hour DriveFuzz margin is stated as 27 in the
text and conclusion [arxiv:2412.13802 §5.2.2] [arxiv:2412.13802 §7]. The
counts in the same results give 35 − 18 = 17 [arxiv:2412.13802 §5.2.2]. Both
sides are cited here and not resolved.

## Papers

- [SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems](/papers/2412.13802): genetic fuzzer for ADS in CARLA with a Transformer violation score and distance-guided mutation.