Moving Forward: A Review of Autonomous Driving Software and Hardware Systems

In plain words

A self-driving car has to see the road with several sensors and understand what it sees with large neural networks. Running those networks fast enough, and on little power, is hard. This paper is a survey of that whole problem. It goes through the sensors a car carries, the datasets and simulators used to test it, the software that turns sensor data into driving decisions, and the chips that run that software.

It describes two styles of driving software. Modular systems split the job into steps such as spotting objects, working out where the car is, predicting what other road users will do, planning a path and controlling the wheels. End-to-end systems use one model that takes sensor data and outputs driving commands.

The authors then take three end-to-end models and run them on four computers, from an embedded board to a desktop with a high-end graphics card. A CPU alone rarely reaches the speed needed, graphics processors are much faster, and moving data to and from memory is a recurring bottleneck. The closing argument is that future car chips will need a mix of general and task-specific hardware, and may need to do some computation close to the memory that holds the data.

TL;DR

• A survey of autonomous-driving software and hardware, plus a benchmark of three end-to-end driving models (TransFuser, InterFuser, MILE) on four CPU/GPU systems §I tab. VII.
• Author-reported: a CPU alone rarely reaches the 10 FPS minimum, and GPUs are faster and more efficient per watt §V-A fig. 8a fig. 8b.
• The authors argue for heterogeneous, task-specific chips and processing-in-memory §V-B §V-C §V-D.
• The benchmark protocol is not described in the text (batch size, precision, timing, power source), so the numbers cannot be reproduced from the paper.
• No official code. Use this card for context and the tables, not as an implementation spec.

Problem

The paper asks what hardware higher levels of driving automation (levels 4 and 5 in the SAE taxonomy) will need, given that the software stack relies on machine learning models with large compute and data-movement demands §I §V-A. Its premise is that off-the-shelf CPU and GPU systems may not scale to that level, and that the mix of tasks and layers in these models calls for more specialized hardware §V-B. It does not define a new algorithm to solve this.

Method

The paper has two parts. It is a survey with one measurement study. It contains no algorithm, no equations and no training procedure.

1. Survey of components. Input sensors (cameras, LiDAR, radar, GPS, IMU) and the sensor setups of five reference systems §II-A tab. I. Datasets (PASCAL VOC, ImageNet, COCO, CIFAR, Cityscapes, KITTI, Waymo open dataset) and simulators (Apollo, Autoware, CARLA) §II-B.
2. Software. Generic modular stacks with perception, localization, prediction, planning and control §III-A, with object detection results in tab. II and Apollo's ML models in tab. III. End-to-end stacks: imitation learning, reinforcement learning and LLM-based approaches §III-B. Tesla's system, described from a public presentation §III-C fig. 6.
3. Hardware survey. GPUs §IV-A, FPGAs §IV-B and SoCs (Tesla FSD, NVIDIA DRIVE and Jetson Orin, Mobileye EyeQ6) §IV-C, compared in tab. IV.
4. Benchmark. Three end-to-end models, TransFuser, InterFuser and MILE, run on four systems §V-A tab. VI tab. VII. Output metrics are frames per second (FPS) fig. 8a and performance per watt fig. 8b. The minimum target is 10 FPS. Arithmetic intensity of sample InterFuser layers is shown in fig. 9 §V-B.
5. Accelerator outlook. Task heterogeneity, layer heterogeneity, and the balance between specialization and flexibility §V-B. Processing-in-memory, with the Neurocube, TETRIS, Ambit, SIMDRAM and DrAcc designs described as examples §V-C fig. 10. A closing outlook on heterogeneous future SoCs §V-D and the conclusion §VI.

The benchmark, as the paper describes it. The paper gives no code or procedure for these steps, so the unstated parts are marked.

# Benchmark as described in the text  [§V-A] [fig. 8]
# Input:  models (3); systems (4), see tab. VII
# Output: FPS and performance per watt for each pair
def benchmark(models, systems):
    for model in models:
        for system in systems:
            # Batch size, resolution, precision and runtime: not stated
            fps = measure_fps(model, system)
            # Power source (measured or TDP): not stated  [fig. 8b]
            efficiency = fps / power(system)
            # Compare against the 10 FPS minimum  [§V-A]
            meets_minimum = fps >= 10
    return results

Section numbers follow the compiled structure. Table and figure numbers follow their order of appearance in the LaTeX source, using IEEE numbering (Roman for tables, Arabic for figures). Numbering was not checked against a compiled PDF.

Hyperparameters

The paper has no training hyperparameters. The table below lists the settings of the benchmark.

Results

Author-reported. The paper has no training or accuracy results of its own. The benchmark results are throughput and efficiency on four systems §V-A.

Object detection, as surveyed (mAP or AP and FPS as reported by the original works, not measured by the authors) tab. II:

Benchmark models tab. VI:

Throughput and efficiency across the four systems fig. 8a fig. 8b. The text gives no numbers for these; the values below are read by eye from the bar charts and are approximate. The figure labels CPU_1 to CPU_4 and GPU_1 to GPU_4, and the paper does not map them to the four systems of tab. VII.

The text states that none of the CPUs alone meets the 10 FPS minimum, except one with borderline results, and that GPUs are more efficient per watt even where their TDP is higher §V-A.

Paper vs code

No official code.

Searched: the LaTeX source contains no code or project links; the arXiv abstract page has no comments field and no links; a GitHub repository search for "Moving Forward autonomous driving hardware" and for the benchmark model names returned no results. No code repository for the benchmark was released with the paper that could be checked.

Pitfalls

• The benchmark cannot be reproduced from the text. The batch size, input resolution, numeric precision, runtime, timing method and power measurement are not stated §V-A fig. 8a fig. 8b. Treat the FPS and FPS-per-watt values as relative trends, not absolute speeds on your hardware.
• Device labels are not mapped. Figure 8 uses CPU_1 to CPU_4 and GPU_1 to GPU_4; the paper does not say which system each label is. Do not assume that GPU_1 is the Jetson platform without checking the source.
• Power is ambiguous. The efficiency metric is performance per watt §V-A. tab. VII lists TDP per system, but the text does not say whether the power used was measured or taken from TDP.
• The 10 FPS threshold has no source. It is stated as the minimum for self-driving, with no citation §V-A fig. 8a.
• Hatching in Figure 8 may not match the threshold. The caption says hatched bars are below the required level. In the figure, some bars near or above the 10 FPS line appear hatched (for example MILE on GPU_1). Check the hatching against the line before relying on it.
• Table II mixes settings. Its rows use different datasets (VOC2007 for 2D, KITTI for 3D), different metrics (mAP or AP) and different hardware. The FPS column does not say what hardware it was measured on. Do not rank the models from this table.
• Hardware units do not reconcile. tab. IV gives 162.6 TOPS for the Nuvo-6108GC, while §IV-A gives "up to 20.31 TFLOPS" for the same platform. TOPS is not defined in terms of precision.
• Jetson AGX Orin power differs between tables. tab. IV lists TDP 40 W for the Jetson AGX Orin 32GB; tab. VII lists TDP 60 W for System 1, which is the Jetson platform. The text does not reconcile them.
• Table V's Tesla HW4 row has no citation. It lists CPU and NNA but no GPU for HW4. The text does not discuss HW4 in detail §V-A.
• Some citations are plain numbers. The text says "Waymo [5] and Apollo [6]" §IV-A. Those bracketed numbers are left-over references and do not point to the bibliography in a clear way. Cite the systems by author or name.
• The Tesla design is from a public presentation. The diagram in fig. 6 is "derived from 2021 Tesla AI day" §III-C. It is not from a paper and has no measured data.
• Parts of the source are not in the compiled paper. The LaTeX source has commented-out sections (testing and evaluation, and potential accelerators with FPGA and ASIC tables) and an older conference-version file. None of them is included in the compiled text. This card does not use them.
• The 95% market projection is a cited claim. §V-A cites a 2023 paper for the projection that AVs will dominate 95% of the market by 2050. The paper is not checked here.

Lineage

• Builds on: none; the paper is a survey. It benchmarks TransFuser, InterFuser and MILE §V-A, which the text names but does not give arXiv ids for.
• Compared against: the survey compares reference systems (Boss, IARA, Waymo, Apollo, Tesla) on sensors and hardware §II-A tab. I tab. V, without a performance comparison between them.
• Superseded by: not known from this wiki.