---
kind: concept
slug: reward-design
title: Reward design
summary: >-
  Choosing the terms of a reinforcement learning reward and where they come from. Options: hand-set terms with a weighted sum or a priority order, physics-based shaping, potential-based shaping from a task graph, learned rewards from demonstrations, a vision-language model or a world model, and redistribution of a sparse team return. Results are author-reported and not on a common benchmark.
tags: [reinforcement-learning, reward-design, reward-shaping, autonomous-driving, humanoid-locomotion, world-models, multi-agent, survey]
papers: ["2306.06766", "2405.01440", "2410.21795", "2412.10096", "2412.13621", "2412.14779", "2412.15587", "2412.16201", "2512.01996", "2512.03556"]
related: [reward-modeling, rlhf, inverse-reinforcement-learning, hierarchical-reinforcement-learning, reinforcement-learning-evaluation, functional-safety, sim-to-real-transfer, domain-randomization, actor-critic, world-model, model-based-reinforcement-learning, learning-from-demonstration, optimal-transport, multi-agent-reinforcement-learning, credit-assignment, deep-q-learning, proximal-policy-optimization, digital-twin, vision-language-pretraining, robotic-grasp-generation]
status: draft
updated_at: 2026-10-25
---

## Overview

A reinforcement learning agent is shaped by its reward, so the reward is a
design choice. For a driving agent the reward has to cover several goals at
once: avoid collisions, make progress, keep the ride comfortable and obey
traffic rules. These goals can conflict, and the design question is how to
turn them into one number [arxiv:2405.01440 §1] [arxiv:2405.01440 §2].

The review in [arxiv:2405.01440 §2] names four difficulties behind this
problem: driving has several objectives, the right trade-off depends on
context, there is no metric for judging a reward function, and rewards are
often delayed. It also notes that unsafe driving can come from a badly defined
reward. The paper is a literature review, not a method. Its findings are
qualitative and rest on a set of reviewed works whose selection is not
described [arxiv:2405.01440 §3].

The other papers answer the same question in other ways. Some write a short
list of terms by hand [arxiv:2512.01996 §2.2] [arxiv:2412.13621 eq. 4].
Some add terms from radio physics or from a task graph [arxiv:2306.06766
§III-B] [arxiv:2412.10096 eq. 7]. Some learn the reward from data: from
demonstrations by optimal transport [arxiv:2410.21795 eq. 5], from a
vision-language model [arxiv:2412.16201 eq. 14], or from a world model
[arxiv:2512.03556 §3.1]. One splits a sparse episode return across time and
agents [arxiv:2412.14779 §4.2]. The papers use different simulators, robots
and tasks, and most compare only against baselines of their own, so their
numbers do not compare directly. All results below are author-reported.

The page covers the design space these papers map out: the terms, how they
are combined, where the reward comes from, and what the authors propose next.
Cards: [Review of driving reward terms](/papers/2405.01440) (arXiv:2405.01440),
[Humanoid sim-to-real recipe](/papers/2512.01996) (arXiv:2512.01996) and
[Unified reward-observation world model](/papers/2512.03556) (arXiv:2512.03556).

For learning a reward from human judgements instead of writing it by hand, see
[reward modeling](/concepts/reward-modeling). For the same question in language
models, see [RLHF](/concepts/rlhf).

## Variants

The variants are grouped by where the reward terms come from. The first group
writes the terms by hand. The driving review sorts them into categories and
compares ways to combine them, and the later hand-set recipes follow the same
pattern.

### Hand-set terms

#### Driving: categories and aggregation

The driving review does not compare full reward functions from end to end. It
treats a reward as a set of category terms plus an aggregation rule. The
first four subsections below follow that split. The last two describe whole-
reward recipes from the later papers.

#### Safety terms

Two families are described [arxiv:2405.01440 §3.1].

- **Direct collision penalty.** A fixed negative value when a collision occurs
  and zero otherwise [arxiv:2405.01440 eq. 1]. The review notes it ignores
  severity: a slow scrape and a fast impact with a pedestrian get the same
  penalty [arxiv:2405.01440 §3.1].
- **Collision penalty scaled by speed.** The penalty grows with the ego
  vehicle's speed at impact, using a formula with a squared speed and a constant
  offset [arxiv:2405.01440 eq. 2]. The review attributes this form to earlier
  work [arxiv:2405.01440 §3.1].
- **Dense time-to-collision (TTC) term.** A risk term that applies only while
  TTC is below a threshold t [arxiv:2405.01440 eq. 3]. The term f is either a
  constant or the inverse of TTC. The review prefers TTC to distance or headway
  measures, citing a comparison study [arxiv:2405.01440 §3.1].

The review's own recommendation is a dense risk term plus a sparse collision
penalty whose size depends on severity and on the road user hit. It is given in
prose, with no equation and no values [arxiv:2405.01440 §3.1].

#### Progress terms

The review lists these forms [arxiv:2405.01440 §3.2]: a delayed reward at the
goal, a per-step time penalty, a dense reward for distance or speed, a penalty
for deviating from a target speed, rewards for acceleration, and rewards for
overtaking. Distance measured along the route is preferred to straight-line
distance, because straight-line progress can reward moving away from the goal
[arxiv:2405.01440 §3.2].

#### Comfort terms

Penalties on high acceleration, deceleration, jerk and steering. The most
complete definition the review found covers longitudinal and lateral
acceleration and jerk, and it leaves out the rate of change of steering and of
lateral acceleration. Several works omit comfort entirely [arxiv:2405.01440
§3.3].

#### Traffic-rule terms

Rewards for staying in lane, penalties for speeding and for undercutting a
minimum headway, and rules for right-of-way and lane position when turning.
Reviewed works penalise the amount of excess speed rather than enforcing speed
limits. None handles several traffic laws together with relaxations
[arxiv:2405.01440 §3.4].

#### Aggregation

- **Summation.** Each term gets the same weight [arxiv:2405.01440 eq. 4]. The
  review says this is the most common design among the works it reviewed, and
  that it cannot express priorities or resolve conflicts [arxiv:2405.01440 §4.1.1].
- **Weighted summation.** Each term has a weight [arxiv:2405.01440 eq. 5]. The
  weights are set by hand or learned with inverse reinforcement learning from
  expert driving. Switching weight sets by context works for simple scenes but
  needs much expert knowledge [arxiv:2405.01440 §4.1.2]. See
  [inverse reinforcement learning](/concepts/inverse-reinforcement-learning).
- **Lexicographic ordering.** Objectives are ranked and decided in order, with
  no weights. Each objective needs a manual threshold, and equal priorities
  cannot be expressed [arxiv:2405.01440 §4.1.3].

#### Proposed directions

These are proposals in the review. None is implemented or evaluated there.

- **Rulebooks.** A set of rules with a priority order, drawn as a directed graph.
  An action is scored by how much it violates the rules, and the priorities may
  be partly learned from data [arxiv:2405.01440 §5.1].
- **Reward machines.** Finite state machines whose states carry sub-task
  rewards. The review suggests they could encode driving context
  [arxiv:2405.01440 §5.2]. Related: [hierarchical reinforcement learning](/concepts/hierarchical-reinforcement-learning).
- **Automatic reward validation.** A framework that checks a reward function
  before training, possibly building on generation of critical scenarios. The
  only prior approach the review found is a manual set of eight sanity checks,
  which it calls hard to apply to driving [arxiv:2405.01440 §5.3].

#### Minimal task reward for humanoid locomotion

The humanoid recipe uses a short list of terms rather than category sums. The
authors state the reward has fewer than ten terms and keeps only the essential
ones [arxiv:2512.01996 §2.2]. The terms are velocity tracking (linear and
angular), swing-foot height tracking, a penalty for leaving the default pose,
feet penalties (parallel feet, no crossing), an alive reward, torso
uprightness penalties and an action-rate penalty [arxiv:2512.01996 §2.2].

Three training choices carry part of the load. Episodes end on ground contact
by the torso or by non-foot body parts [arxiv:2512.01996 §2.2]. Penalty weights
follow a curriculum that raises them as episode length grows [arxiv:2512.01996
§2.2]. Symmetry augmentation is used [arxiv:2512.01996 §2.2]. The paper does
not state the reward weights [arxiv:2512.01996 §2.2]. For whole-body motion
tracking the authors follow BeyondMimic, with DeepMimic-style termination and
velocity pushes [arxiv:2512.01996 §2.2].

#### Quadruped pipe crossing: seven terms and stage training

The pipe-crossing policy uses seven reward terms [arxiv:2412.13621 §II-C]:
linear velocity tracking with a heading term [eq. 4], yaw-rate tracking
[eq. 5], three energy penalties on torque, torque change and joint
acceleration [eq. 6] [eq. 7] [eq. 8], a contact penalty [eq. 9], and a penalty
on the distance to the pipe centreline [eq. 10]. The fixed weights are 1.5 for
linear velocity, 0.5 for yaw, 10 for contact and 0.3 for the centreline, with
much smaller weights on the energy terms [arxiv:2412.13621 tab. II]. The paper
does not compare these weights with other settings.

Training runs in three stages. Pipe radius is 0.3 to 0.5 m in stage 1, then
0.2 to 0.3 m in stages 2 and 3, and obstacles appear only in stage 3
[arxiv:2412.13621 tab. III]. In simulation, the stage-3 policy succeeds in all
16 runs for 0.2 m and 0.3 m pipes without obstacles, and in 93.75% of runs for
0.2 m pipes with obstacles [arxiv:2412.13621 tab. IV]. On three real PVC pipes
the robot crossed every trial without obstacles. With obstacles it reached
46.5% on one pipe [arxiv:2412.13621 tab. V]. The pipe-radius curriculum is a
terrain curriculum that moves the robot between levels by distance travelled
[arxiv:2412.13621 §II-A1]. It is not a reward curriculum.

#### Dexterous grasp: three reward groups

The reward for the in-hand stage sums three groups, with balance weights
alpha, beta and eta that the paper does not state [arxiv:2412.15587 eq. 6].
The interaction group pulls the fingers toward the functional part of the
object and the palm toward the object [arxiv:2412.15587 eq. 7] [eq. 8]
[eq. 9]. The completion group rewards task progress, which the paper defines
only for the laptop as its opening angle, plus a bonus that shrinks with the
step count [arxiv:2412.15587 eq. 10]. The restriction group penalises arm
motion and arm-link contact with the object [arxiv:2412.15587 eq. 11]
[eq. 12] [eq. 13]. The grasp prior that sets the start pose is the main
contribution, and the reward is a secondary part. In simulation the method
beats DexArt on four tasks, for example 94.17% against 79.17% on the bucket
[arxiv:2412.15587 tab. 1]. The real UR5 tests cover two tasks, with about ten
trials each [arxiv:2412.15587 §4.2].

### Physics and task-graph shaping

#### Radio physics terms

For mmWave indoor navigation, the reward adds three physics terms to a distance
cost: a link-state monotonicity penalty [arxiv:2306.06766 eq. 12], an
angle-of-arrival following penalty [arxiv:2306.06766 eq. 13], and, in the
weakest link state, an SNR-gradient penalty [arxiv:2306.06766 eq. 14]
[§III-B]. The paper's weights are 1.0, 0.5 and 0.3 [arxiv:2306.06766 tab. 2],
and the global reward is grid-searched over [0.5, 2] [arxiv:2306.06766 §IV-B].
The paper says the shaped reward is not potential-based, so it changes the
optimal policy, and it argues this helps transfer [arxiv:2306.06766 §III-B].

On six unseen maps, the normalised path length in the 2+-NLOS link state is
2.46 to 2.82 for this method, against 4.50 to 5.14 for end-to-end RL
[arxiv:2306.06766 tab. 3]. In the ablation, removing the SNR term raises the
2+-NLOS average to 4.62 and removing the link-state term raises it to 3.90,
against 2.60 for the full reward [arxiv:2306.06766 tab. 5]. The released code's
reward has neither term and feeds the policy the ground-truth link-state label,
so these ablations cannot be re-run from the code. See the
[card](/papers/2306.06766) for the differences.

#### Potential-based shaping from a reward machine

A reward machine is a finite graph of sub-goals. The reward-machine paper
infers one from unlabelled RGB demonstrations [arxiv:2412.10096 §4.2]
[§4.4]. Frames are embedded by a pretrained ResNet-50, clustered with DBSCAN,
and each cluster's frame closest to its mean becomes a sub-goal prototype
[arxiv:2412.10096 eq. 3] [eq. 4] [§4.3]. Each prototype gives one machine state
and one proposition, which is true when the feature distance is below a
threshold κ [arxiv:2412.10096 eq. 5] [§4.4].

The shaping potential is γ raised to the number of edges from the state to the
goal [arxiv:2412.10096 eq. 6]. The shaped transition reward adds the usual
potential difference to the sparse reward [arxiv:2412.10096 eq. 7] [§4.4]. A
DQN on the inferred machine performs similarly to one given the true machine
in five simulated block tasks, except Stack-3, where the baseline plateaus
higher [arxiv:2412.10096 §6] [fig. 5]. The paper gives no numeric table. The
threshold κ, the clustering settings and the number of demonstrations are tuned
per task, and the procedure is not described [arxiv:2412.10096 tab. 1]. No code
is released. The driving review also lists reward machines as a future option
[arxiv:2405.01440 §5.2].

### Learned rewards

#### From demonstrations: optimal transport

TemporalOT relabels the agent's trajectories with a reward from optimal
transport against one or two expert videos [arxiv:2410.21795 §3.1]
[eq. 5]. Two changes make the reward order-aware: a cost averaged over
k_c consecutive frames [arxiv:2410.21795 eq. 6], and a band mask around the
diagonal of the transport plan [arxiv:2410.21795 eq. 9]. The paper uses
k_c = 3 and a mask half-width of 10 [arxiv:2410.21795 tab. 4].

Author-reported: averaged over nine Meta-World tasks with two demos, success is
61.1, against 35.9 for ADS and 20.8 for the task reward [arxiv:2410.21795
tab. 1]. The band assumes the expert moves at a similar speed. With demos
sampled three times faster, Door-open success falls from 78.4 to 49.4
[arxiv:2410.21795 tab. 3] [§3.2.3]. Each reward needs a Sinkhorn solve per
trajectory and per demo, about 1 h on state input and 3 h on pixels on one
RTX 3090 [arxiv:2410.21795 app. B.4]. The authors' code has no license file,
so this page does not copy from it.

#### From a vision-language model: CLIP bonus

CLIP-RLDrive adds a bonus from an image-text model to the simulator reward. Each
frame is scored against three action sentences by cosine similarity
[arxiv:2412.16201 eq. 12], and the bonus is added with weight W_c = 1.2
[arxiv:2412.16201 eq. 14] [§2.4]. The paper's Algorithm 1 gives a different
rule: the bonus is added only when the agent's action matches the action CLIP
suggests [arxiv:2412.16201 alg. 1].

Author-reported on an unsignalized intersection: CLIP-DQN succeeds in 96% of
episodes with one other vehicle, against 70% for DQN. With six vehicles it
reaches 72%, against 50% for DQN [arxiv:2412.16201 tab. 2]. CLIP-PPO is below
plain PPO at every vehicle count, with 38% against 78% at one vehicle
[arxiv:2412.16201 tab. 2]. The card records that the released code gives the
no-CLIP baseline the same bonus, so that comparison cannot be checked from the
repository. See the [card](/papers/2412.16201).

#### From a world model

Instead of writing terms, this variant reads the reward from a learned world
model. A text model imagines a goal frame from the instruction. The reward
then has two parts: a dense term, the LPIPS distance between the current frame
and that goal frame [arxiv:2512.03556 eq. 5], and a sparse term, one when the
action model's predicted done signal reaches a threshold [arxiv:2512.03556
eq. 7] [arxiv:2512.03556 eq. 8].

The paper is not consistent on how the two parts combine. Equation 8 adds them
without weights, while Algorithm 1 mixes them with a weight alpha and 1 − alpha
[arxiv:2512.03556 eq. 8] [arxiv:2512.03556 alg. 1]. The value of alpha is not
stated. LPIPS is a distance, so a lower value means more similar frames. The
equations add the dense term as written, and the text calls it a similarity
[arxiv:2512.03556 eq. 5] [arxiv:2512.03556 §3.1]. An implementation that
follows the equations literally would reward moving away from the goal, so the
sign has to be fixed by the implementer.

Author-reported: policies trained with this reward reach 84.1% on unseen
pick-and-place scenes with OpenVLA, against 54.5% with the simulator's
rule-based reward [arxiv:2512.03556 tab. 1]. In-domain differences are small,
and the table gives single runs with no error bars [arxiv:2512.03556 tab. 1].
All evaluation is in simulation [arxiv:2512.03556 §5].

### Redistributing a sparse team return

TAR² trains a reward model that splits one episodic team return into per-step,
per-agent rewards. The temporal weights and the agent weights are non-negative,
each set sums to 1, and the per-agent reward is their product times the return
[arxiv:2412.14779 §4.1] [§4.2]. The paper argues that this redistribution is
potential-based shaping and so keeps the optimal policy [arxiv:2412.14779
§4.3]. The proof sets the discount to 1, while the paper's own setting uses a
discount [arxiv:2412.14779 §3.1] [§4.3].

The learners are IPPO and MAPPO [arxiv:2412.14779 §5.1]. The only evaluation is
one SMACLite scenario, 5m_vs_6m, shown as a figure with no numbers
[arxiv:2412.14779 §5.2] [fig. 1]. The paper's motivation is that a sparse
team reward makes the advantage noisier as the number of agents grows
[arxiv:2412.14779 §3.4]. The paper says the reward model is warmed up for 2000
episodes before use [arxiv:2412.14779 §6]. The released code's default is 200
episodes, and the card lists other differences.

## When to use which

The driving review gives no experiments, so its rows state constraints and what
the review says about each option. It does not rank the options by performance.
The later rows carry the reported results of their papers. Those results are
author-reported and come from different tasks, so they are not ranked against
each other.

| Situation | Use | Why |
|---|---|---|
| Quick baseline, no reward-specific data | Summation [arxiv:2405.01440 eq. 4] | Simplest form, but cannot express priorities or resolve conflicts [arxiv:2405.01440 §4.1.1] |
| Trade-offs known, weights can be tuned | Weighted summation [arxiv:2405.01440 eq. 5] | Hand tuning or IRL; the review gives no weighting guideline [arxiv:2405.01440 §4.1.2] |
| Expert driving data available | Weights learned by IRL [arxiv:2405.01440 §4.1.2] | Reduces hand tuning; still needs expert data |
| One objective must dominate another | Lexicographic ordering [arxiv:2405.01440 §4.1.3] | Strict order, but needs a threshold per objective and cannot give equal priority |
| Collisions only, severity irrelevant | Direct collision penalty [arxiv:2405.01440 eq. 1] | Simple; the review says it ignores severity [arxiv:2405.01440 §3.1] |
| Early warning before collisions | Dense TTC term [arxiv:2405.01440 eq. 3] | The review prefers TTC; the threshold t is not given [arxiv:2405.01440 §3.1] |
| Comfort needed, safety is critical | Comfort terms, checked against safety | Hard-braking penalties can conflict with safety [arxiv:2405.01440 §3.3] |
| Several contexts or use cases | No validated option | No reviewed work is context-aware, and no transition mechanism exists yet [arxiv:2405.01440 §4.2] |
| Economic goals such as fuel or cost | No reviewed option | No reviewed work considers economic terms [arxiv:2405.01440 §4.3] |
| Reward must be checked before training | No validated option | Only a manual set of sanity checks exists [arxiv:2405.01440 §5.3] |
| Humanoid locomotion in a massively parallel simulator | Short term list with curriculum-ramped penalties [arxiv:2512.01996 §2.2] | The authors report walking policies in 15 minutes on one RTX 4090 in simulation [arxiv:2512.01996 §3.1]; weights are not in the paper |
| Humanoid whole-body motion tracking | Tracking reward after BeyondMimic [arxiv:2512.01996 §2.2] | Hardware deployment is shown for this task only, on G1 [arxiv:2512.01996 §3.2] |
| Quadruped crossing a narrow pipe, simulator with privileged terrain input | Seven fixed-weight terms with a terrain curriculum [arxiv:2412.13621 tab. II] | Real tests use few trials; the paper reports 46.5% with obstacles on one pipe [arxiv:2412.13621 tab. V] |
| Dexterous grasp of an articulated part, known object geometry | Grasp prior for the start pose, three reward groups [arxiv:2412.15587 §3.4.3] | Beats DexArt in simulation [arxiv:2412.15587 tab. 1]; the balance weights are not stated [arxiv:2412.15587 eq. 6] |
| Wireless or radio signal with known physics, unseen maps | Physics terms added to the reward [arxiv:2306.06766 eq. 10] | Author-reported path-length gains on six unseen maps [arxiv:2306.06766 tab. 3]; the released code omits the terms (see [card](/papers/2306.06766)) |
| Discrete sub-goals visible in demonstrations, no labels | Reward machine inferred from demos, potential shaping [arxiv:2412.10096 eq. 6] [eq. 7] | Simulation only; no numeric results table; κ and clustering tuned per task [arxiv:2412.10096 tab. 1] |
| Few demonstrations, order of motion matters, no task reward | Order-aware optimal-transport reward [arxiv:2410.21795 eq. 6] [eq. 9] | Author-reported 61.1 against 35.9 for ADS on nine tasks; drops with faster demos [arxiv:2410.21795 tab. 3] |
| Task described in text, pretrained image-text model, no labelled reward | CLIP bonus with a fixed weight [arxiv:2412.16201 eq. 14] | One intersection scenario; the bonus did not help PPO [arxiv:2412.16201 tab. 2] |
| Task with no simple reward, a trained world model available | Learned goal-distance and done reward [arxiv:2512.03556 §3.1] | Simulation results only; the world model cost about 96 hours of training on an H20 cluster [arxiv:2512.03556 §4.2] |
| Only an episodic team outcome, several agents, single-agent learner | Redistribution of the return over time and agents [arxiv:2412.14779 §4.2] | Evaluated on one scenario, as a figure only; the policy-preservation proof uses discount 1 [arxiv:2412.14779 §4.3] |
| A trusted simulator reward exists | Keep the simulator reward | The learned reward's in-domain gains are small [arxiv:2512.03556 tab. 1] |

## Implementation sketch

The driving review gives the aggregation forms and the safety terms as
equations. The learned-reward sketch transcribes the paper's equations 5 and 7
and its Algorithm 1 combination. The other sketches transcribe the potential
shaping, the CLIP bonus and the redistribution forms. The sketches below are our
own transcription of those forms, not code from the papers. The papers give no
numeric values for x, t, the weights, the discount or the thresholds.

```pseudocode
# Direct collision penalty  [arxiv:2405.01440 eq. 1]
# x: negative penalty chosen by the designer; value not stated
def collision_term(collided, x):
    if collided:
        return x
    return 0.0

# Dense TTC term  [arxiv:2405.01440 eq. 3]
# t: threshold, not stated; f: constant or 1/TTC (guard TTC = 0 yourself)
def ttc_term(ttc, t, f):
    if ttc < t:
        return f(ttc)
    return 0.0

# Summation  [arxiv:2405.01440 eq. 4]
def reward_sum(r_safety, r_progress, r_comfort, r_rules):
    return r_safety + r_progress + r_comfort + r_rules

# Weighted summation  [arxiv:2405.01440 eq. 5]
# weights: hand-tuned or IRL-learned; values not stated
def reward_weighted(terms, weights):
    return sum(w * r for w, r in zip(weights, terms))

# Potential-based shaping  [arxiv:2412.10096 eq. 6] [eq. 7]
# gamma: discount, not stated in the paper; use the TD-target value
def potential_shaped(sparse, d_u, d_u_next, gamma):
    phi = gamma ** d_u
    phi_next = gamma ** d_u_next
    return sparse + gamma * phi_next - phi

# Additive CLIP bonus  [arxiv:2412.16201 eq. 12] [eq. 14]
# w: 1.2 in the paper; Algorithm 1 gates the bonus on agreement instead
def clip_bonus(r_basic, r_clip, w):
    return r_basic + w * r_clip

# Sparse done term  [arxiv:2512.03556 eq. 7]
# theta_sps: threshold, not stated
def sparse_term(done_pred, theta_sps):
    if done_pred >= theta_sps:
        return 1.0
    return 0.0

# Dense goal-distance term  [arxiv:2512.03556 eq. 5]
# lpips: lower means more similar; eq. 5 gives no sign, so decide one
def dense_term(frame, goal, lpips):
    return lpips(frame, goal)

# Learned-reward combination  [arxiv:2512.03556 alg. 1]
# alpha: weight, not stated; eq. 8 has no weights
def reward_learned(sparse, dense, alpha):
    return alpha * sparse + (1.0 - alpha) * dense

# Redistribution of an episodic return  [arxiv:2412.14779 §4.2]
# temporal_w: (T,), sums to 1; agent_w: (N, T), sums to 1 over agents
def redistribute(episode_return, temporal_w, agent_w):
    return agent_w * temporal_w * episode_return   # (N, T), sums to return
```

## Contradictions

The papers do not test each other's claims. Where they disagree, both sides
are stated and the choice is left to the implementer.

**Potential-based or not.** The TAR² paper argues that its redistribution is
potential-based and so keeps the optimal policy [arxiv:2412.14779 §4.3]. The
physics-terms paper states that its shaped reward is not potential-based and
changes the optimal policy [arxiv:2306.06766 §III-B]. The reward-machine paper
uses a potential-based term [arxiv:2412.10096 eq. 7]. The TAR² proof uses
discount 1, which the paper's own setting does not use [arxiv:2412.14779 §4.3].

**Fixed weights.** The driving review argues that fixed weighted or summed
terms cannot express priorities or resolve conflicts [arxiv:2405.01440 §4.1.1].
The humanoid recipe uses penalty weights that change over a curriculum and
reports working locomotion and tracking policies [arxiv:2512.01996 §2.2]
[arxiv:2512.01996 §3.1]. The pipe-crossing paper uses fixed weights and reports
stage-wise success [arxiv:2412.13621 tab. II] [tab. IV]. None of these papers
compares against other weightings. The papers address different robots and
tasks, so each statement stands as its authors make it.

**CLIP gating.** The CLIP paper's equation adds the bonus on every step, with
weight 1.2 [arxiv:2412.16201 eq. 14]. Its Algorithm 1 adds it only when the
action agrees with CLIP [arxiv:2412.16201 alg. 1]. An implementation must pick
one. The card records a third rule in the released code.

**Inside the world-model paper.** The text calls the LPIPS term a similarity,
but LPIPS measures distance, and the equations add the term as written
[arxiv:2512.03556 §3.1] [arxiv:2512.03556 eq. 5]. The paper also uses two
different forms of the reward weight [arxiv:2512.03556 eq. 8] [arxiv:2512.03556
alg. 1]. Implementers should pick one form and state it.

**Inside the driving review.** The text says speed can enter the safety penalty
exponentially, but the displayed formula uses a square [arxiv:2405.01440 §3.1]
[arxiv:2405.01440 eq. 2]. The review's §3 names five categories, one of them
about model performance, but analyses four [arxiv:2405.01440 §3]. Implementers
should pick one form and state it.

**Paper against released code.** The physics-terms paper's own reward and its
released reward differ. The code has no link-state or SNR term, although the
paper's ablations credit those terms [arxiv:2306.06766 tab. 5]. The
[card](/papers/2306.06766) lists the other differences, including the
weights, which differ from the paper's table.

## Papers

- [Review of driving reward terms](/papers/2405.01440): reviews reward terms for RL driving agents, compares aggregation forms, and proposes rulebooks, reward machines and reward validation without experiments.
- [Humanoid sim-to-real recipe](/papers/2512.01996): a short humanoid reward with curriculum-ramped penalties, trained with off-policy learners on massively parallel simulation; author-reported 15-minute walking policies.
- [Unified reward-observation world model](/papers/2512.03556): a reward read from a learned world model, a goal-frame distance plus a done signal; author-reported gains over the simulator reward on unseen scenes.
- [Digital twin-enhanced wireless navigation](/papers/2306.06766): three radio-physics terms added to a PPO navigation reward; author-reported path-length gains on unseen maps, with the released code omitting the terms.
- [Reward machine inference](/papers/2412.10096): infers a reward machine from unlabelled demonstrations and shapes the DQN reward with a potential; results are figures only.
- [Temporal optimal transport reward](/papers/2410.21795): an order-aware optimal-transport reward from one or two demos; author-reported success on nine Meta-World tasks.
- [Credit assignment for sparse multi-agent RL](/papers/2412.14779): redistributes a sparse team return into per-step, per-agent rewards; one scenario, shown only as a figure.
- [Quadruped pipe crossing](/papers/2412.13621): a seven-term fixed-weight reward with a terrain curriculum; author-reported simulation results and few real-pipe trials.
- [Dexterous manipulation with a grasp prior](/papers/2412.15587): a grasp-based start pose and a three-group reward for PPO; author-reported gains over DexArt in simulation.
- [CLIP reward shaping for driving](/papers/2412.16201): adds a CLIP image-text bonus to DQN and PPO rewards at an unsignalized intersection; author-reported gains for DQN only.