A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving

In plain words

A reinforcement learning agent learns by chasing a score, the reward. For a self-driving car that score has to balance several goals at once: do not crash, make progress, ride smoothly and obey traffic rules. These goals often pull in different directions, and it is hard to decide how much each one should count.

This paper collects the scores that recent research has used for driving, sorts every term into four groups (safety, progress, comfort, traffic rules), and looks at what each group gets right and wrong. Its main complaints are that the terms are usually added together with fixed weights, that they ignore the driving context, and that nobody has a systematic way to check a reward before training on it. The paper suggests three directions: rulebooks, which are sets of rules with a priority order; reward machines, which switch between sub-goal rewards; and automatic checks of reward functions. It runs no experiments of its own.

TL;DR

• A review, not a method: it groups the reward terms of reviewed RL driving papers into safety, progress, comfort and traffic-rule categories §3.
• Common designs are summed or hand-weighted terms, with no context awareness and no priority handling between objectives §4.1, §4.2.
• Proposes rulebooks §5.1, reward machines §5.2 and an automatic reward validation framework §5.3. None is implemented or evaluated in the paper.
• Use it to see the reward-term design space and its known failure modes before writing a driving reward; do not treat it as a tested reward recipe.

Problem

Reinforcement learning is a leading approach for end-to-end driving, and its behaviour is set by the reward function §1. Designing that function is hard for four reasons the paper attributes to prior work: driving is multi-objective, the right trade-offs depend on context such as region, weather and road type, there are no performance metrics for judging a reward function, and rewards are often delayed §2. Badly defined rewards can produce unsafe driving §2. The paper argues that the design phase gets too little attention §2. It focuses on reward functions, although its limitations also apply to objective functions in modular stacks §2.

Method

The paper is a literature review. Its procedure, as stated in the compiled text:

1. Break each reviewed reward function into individual terms §3.
2. Assign each term to a category: safety, progress, comfort or traffic-rule conformance §3. The §3 introduction also names a fifth category, "related to model performance", but the analysis covers four. Terms that only help training, such as exploration bonuses, are excluded because they do not describe driving objectives §3.
3. For each category, compare the formulations, their strengths and their limitations §3.1 to §3.4.
4. Analyse the structure of the whole reward: how terms are aggregated, and how far they depend on driving context §4.
5. Propose future directions: rulebooks §5.1, reward machines §5.2, and a validation framework §5.3.

The compiled paper has no tables and no figures. Its LaTeX source contains figures, but they sit inside comment blocks and are not compiled.

Safety §3.1. The paper describes two families. A direct method penalises collisions. A situational method estimates risk and rewards safe margins.

The simplest direct form is a conditional penalty: a fixed negative value x when a collision happens, and zero otherwise eq. 1. The paper points out that it ignores severity, so a slow scrape and a high-speed pedestrian impact get the same penalty §3.1.

r_{safety} = \begin{cases} x & \text{if there is a collision} \\ 0 & \text{otherwise} \end{cases} eq. 1

A variant scales the penalty with the ego vehicle's speed at the collision, which the paper attributes to earlier work §3.1. Its formula is quadratic in speed with a constant offset eq. 2.

r_{safety} = -x \times (velocity^2 + 0.5) eq. 2

Dense alternatives penalise risk before a collision happens. The paper favours time-to-collision (TTC) over distance or headway, citing a comparison study §3.1. With a threshold t, a term f(TTC) applies while TTC is below t eq. 3. f is either a constant or the inverse of TTC §3.1.

r_{safety} = \begin{cases} f(TTC) & \text{if } TTC < t \\ 0 & \text{otherwise} \end{cases} eq. 3

The paper's own recommendation combines a dense risk term (TTC or headway) with a sparse collision penalty whose size depends on severity and on the type of road user hit §3.1. This recommendation is given in prose only; it has no equation and no values §3.1.

Progress §3.2. Forms reviewed: a delayed reward for reaching the goal; a per-step time penalty; a dense reward proportional to distance travelled or to speed; a penalty for deviating from a target speed, usually the speed limit; rewards for acceleration; and rewards for overtaking. Distance-based proxies can reward moving away from the goal. Measuring distance along the road route is preferred to straight-line distance §3.2. The paper cites Knox et al. for a case where a progress penalty over a long wait exceeds the collision penalty, so the agent prefers to crash into a static obstacle rather than wait §3.2.

Comfort §3.3. Penalties on high acceleration and deceleration, on jerk, on steering angle, and on counter-steering. Hard-braking penalties can conflict with safety. The most complete comfort definition the authors found covers right-of-way, longitudinal and lateral acceleration and jerk. It omits the rate of change of steering angle and of lateral acceleration §3.3. Several works leave comfort out entirely §3.3.

Traffic-rule conformance §3.4. Rewards for staying in lane, penalties for speeding and for undercutting a minimum headway, and rules for right-of-way and for lane position when turning. Reviewed works do not enforce speed limits strictly; they penalise the amount of excess speed §3.4. No reviewed work handles several traffic laws at once with relaxations §3.4.

Aggregation §4.1. Summation eq. 4 gives every term the same weight, and the paper notes this cannot express priorities or resolve conflicts §4.1.1.

r = r_{safety} + r_{progress} + r_{comfort} + r_{rules} eq. 4

Weighted summation eq. 5 assigns each term a weight. Weights are tuned by hand or learned with inverse reinforcement learning (IRL) from expert driving §4.1.2. Switching between reward sets by context works in simple scenes but needs much expert knowledge §4.1.2.

r = w_{safety} r_{safety} + w_{progress} r_{progress} + w_{comfort} r_{comfort} + w_{traffic\ rules} r_{rules} eq. 5

Lexicographic ordering ranks objectives and decides in that order, with no weights. It needs a manual threshold per objective and cannot represent equal priorities §4.1.3.

Context §4.2 and economics §4.3. Reviewed works tune rewards to one use case, mostly urban driving, and none make the reward context-aware §4.2. The paper suggests a hybrid of use-case terms and general terms, which needs a transition mechanism between use cases; the authors say none exists yet §4.2. No reviewed work considers economic terms such as fuel efficiency or cost §4.3.

Future work §5. Rulebooks §5.1 are a tuple of rules and a priority order, and can be drawn as a directed graph §5.1. A driving action is scored by how much it violates the rules, with priorities applied, and the priorities may be partly learned from data §5.1. Reward machines §5.2 are finite state machines whose states carry sub-task rewards, and they could encode driving context §5.2. The authors call for an automatic framework that validates reward functions, possibly building on critical-scenario generation §5.3. They note that the only validation approach they found is a manual set of eight sanity checks, which is hard to apply to driving §5.3.

The reward forms from the paper, transcribed as pseudocode (not code from the authors; no official code exists):

# Conditional collision term  [§3.1] [eq. 1]
# x: manually defined negative penalty; value not stated  [§3.1]
def collision_term(collided, x):
    if collided:
        return x
    return 0.0

# Collision term scaled by ego speed  [§3.1] [eq. 2]
# Text says speed "can be" applied exponentially; eq. 2 uses a square.
def collision_term_speed(x, velocity):
    return -x * (velocity ** 2 + 0.5)

# Dense time-to-collision term  [§3.1] [eq. 3]
# f: constant or 1/TTC (the paper's two options); t: threshold, not stated.
def ttc_term(ttc, t, f):
    if ttc < t:
        return f(ttc)
    return 0.0

# Plain sum of the four categories  [§4.1.1] [eq. 4]
def reward_sum(r_safety, r_progress, r_comfort, r_rules):
    return r_safety + r_progress + r_comfort + r_rules

# Weighted sum  [§4.1.2] [eq. 5]
# Weights are tuned by hand or learned with IRL; values not given.
def reward_weighted(terms, weights):
    total = 0.0
    for term, weight in zip(terms, weights):
        total = total + weight * term
    return total

Hyperparameters

The paper trains nothing, so it gives no training hyperparameters. The symbols in its equations, with the values the paper gives:

Results

The paper reports no experiments and no metrics of its own. The findings below are the authors' qualitative claims from reviewing the literature. They depend on a selection of papers whose procedure is not stated (see Pitfalls).

Paper vs code

No official code.

The paper is a review, so it has no implementation to compare with. Searches: the arXiv abstract page lists no code or project link (fetched); the LaTeX source contains no URL; a GitHub repository search for "reward functions autonomous driving" returned no repository owned by the authors or their institutions.

Pitfalls

• Category count. §3 names five categories, including model performance, but the analysis covers four and drops exploration-type terms §3. Treat the four driving categories as the scope.
• Selection is not described. The compiled text does not say how the reviewed reward functions were found or chosen. Statements such as "none of the state-of-the-art RL approaches" and "to the best of our knowledge" §3.1 therefore cannot be checked from the paper.
• Citations are numbers. Most claims point to numbered references (for example [r8], [r12]). Their titles are only in the paper's bibliography, so check the bibliography before relying on one.
• Eq. 2 versus text. The text says speed "can be" introduced exponentially §3.1, but the displayed equation uses a square eq. 2. The constant 0.5 and the sign convention are not explained. Choose the form explicitly.
• Eq. 3 at zero TTC. The inverse-TTC option is undefined when TTC is zero eq. 3. The paper gives no clamp or floor (this is an inference from the formula, not a stated detail). The threshold t is also not given.
• Recommended safety reward is prose only. The combined dense-plus-sparse safety term §3.1 has no equation, no weights and no severity levels. An implementer must invent these, and the paper does not say how.
• Summation versus weights. Eq. 4 and eq. 5 are presented as the baseline and its weighted fix. The paper gives no rule for the weights. Its point that summation weights safety and progress equally §4.1.1 holds only for eq. 4 as written.
• Progress terms can reward the wrong motion. Distance-based progress rewards the agent for moving backwards §3.2. The paper's preferred fix, route distance, is only suggested §3.2.
• Penalties can conflict. A progress or time penalty can outweigh a collision penalty, so the agent prefers a crash to waiting §3.2. Hard braking penalties can push against safety §3.3. Check the magnitudes.
• Context and economics are gaps, not solved problems. No transition mechanism between use cases exists in the field, per the authors §4.2, and economic terms are absent from the reviewed rewards §4.3.
• Figures. The source's figures are inside comment blocks and are not in the compiled paper, so no figure citations exist for this card.

Lineage

• Builds on: Knox et al., "Reward (mis) design for autonomous driving", the source of the four design challenges §2 and the eight sanity checks for reward validation §5.3.
• Builds on: rulebooks from Censi et al. §5.1, with the rule-based optimal-control work of Xiao et al. §5.1.
• Builds on: reward machines (Icarte et al., "Reward machines: Exploiting reward function structure in reinforcement learning") §5.2.
• Compared against: the reviewed reward functions of RL driving papers, which are not named individually in the text §3.
• Superseded by: not known from content in this wiki.