Fuzz testing

Overview

Fuzz testing is a search over test inputs. The system under test is run on a candidate input, the run is scored for signs of failure, and promising inputs are kept and varied. For self-driving software the input is a whole scenario: routes for the ego vehicle and other traffic, pedestrians and weather 2412.13802 §4.2. The problem is that the space of scenarios is very large, so the question is which ones to run next 2412.13802 §1.

The shared core is a population of scenarios that is scored by feedback, selected, crossed over and mutated 2412.13802 §4.2. What differs between designs is the feedback, the score and how the variation is done. The only card in this wiki so far covers one such design; see Variants.

Variants

Simulation-feedback genetic fuzzing (SimADFuzz)

The fuzzer runs each scenario in CARLA and reads per-vehicle position, speed, acceleration and heading at every frame 2412.13802 §4.1 tab. 2. A Transformer encoder reads that sequence and outputs a violation score that is one of four fitness objectives. The others are a road-feature classifier (SDC-Scissor), the minimum distance to the ego vehicle and the count of unique violations. These are combined with NSGA-II 2412.13802 §4.2.1. The design changes two parts of earlier genetic fuzzers. The score reads the whole run instead of one aggregate, and mutation removes vehicles that are stuck or moving away from the ego vehicle instead of changing them at random 2412.13802 §3 alg. 1. Card: SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems.

Author-reported results use InterFuser on Town03 with 6-hour runs. It finds 35 unique violations, against 3 for AV-Fuzzer, 18 for DriveFuzz and 26 for TM-Fuzzer 2412.13802 §5.2.2 fig. 8. Its first collision comes at 17 minutes, against 21 minutes for TM-Fuzzer 2412.13802 §5.2.2.

When to use which

The choice depends on what you can run and what you can build. The only design with evidence here is the simulation-feedback one, so the rows below are about its constraints.

Implementation sketch

Language-agnostic outline of the loop in the simulation-feedback design. Where the variants differ, the choices are marked. Parameters are the paper's 2412.13802 §5.1.2.

# Seed a population of random scenarios  [arxiv:2412.13802 §5.1.2]
population = [random_scenario() for _ in range(20)]

# Repeat until the time budget is spent  [arxiv:2412.13802 §5.2]
while not budget_spent():
    # Run each scenario and record per-vehicle states per frame  [§4.1]
    traces = [simulate(s) for s in population]

    # Score each scenario on several objectives, e.g. a learned score,
    # a road score, minimum distance and unique violations  [§4.2.1]
    scores = [objectives(s, t) for s, t in zip(population, traces)]

    # Make offspring; each operator runs with probability 0.5  [§5.1.2]
    children = []
    for pair in pick_parents(population, scores):
        # Swap routes between parents, keeping the ego route  [§4.2.2]
        child = crossover(pair)
        # Remove stuck or leaving NPCs and redraw the rest  [§4.2.3]
        child = mutate(child)
        children.append(child)

    # Keep the best of parents and children under NSGA-II  [§4.2.1]
    population = nsga2_select(population + children, scores)

# Report violations found in the run  [§4.3]
return unique_violations(population)

Variants differ in the objectives used (this design uses four 2412.13802 §4.2.1), in the mutation rule 2412.13802 §4.2.3, and in the run length 2412.13802 §5.2.

Contradictions

Within the one card so far: the 6-hour DriveFuzz margin is stated as 27 in the text and conclusion 2412.13802 §5.2.2 2412.13802 §7. The counts in the same results give 35 − 18 = 17 2412.13802 §5.2.2. Both sides are cited here and not resolved.

Papers

• SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems: genetic fuzzer for ADS in CARLA with a Transformer violation score and distance-guided mutation.