Reward design

Overview

A reinforcement learning agent is shaped by its reward, so the reward is a design choice. For a driving agent the reward has to cover several goals at once: avoid collisions, make progress, keep the ride comfortable and obey traffic rules. These goals can conflict, and the design question is how to turn them into one number 2405.01440 §1 2405.01440 §2.

The review in 2405.01440 §2 names four difficulties behind this problem: driving has several objectives, the right trade-off depends on context, there is no metric for judging a reward function, and rewards are often delayed. It also notes that unsafe driving can come from a badly defined reward. The paper is a literature review, not a method. Its findings are qualitative and rest on a set of reviewed works whose selection is not described 2405.01440 §3.

The other papers answer the same question in other ways. Some write a short list of terms by hand 2512.01996 §2.2 2412.13621 eq. 4. Some add terms from radio physics or from a task graph [arxiv:2306.06766 §III-B] 2412.10096 eq. 7. Some learn the reward from data: from demonstrations by optimal transport 2410.21795 eq. 5, from a vision-language model 2412.16201 eq. 14, or from a world model 2512.03556 §3.1. One splits a sparse episode return across time and agents 2412.14779 §4.2. The papers use different simulators, robots and tasks, and most compare only against baselines of their own, so their numbers do not compare directly. All results below are author-reported.

The page covers the design space these papers map out: the terms, how they are combined, where the reward comes from, and what the authors propose next. Cards: Review of driving reward terms (arXiv:2405.01440), Humanoid sim-to-real recipe (arXiv:2512.01996) and Unified reward-observation world model (arXiv:2512.03556).

For learning a reward from human judgements instead of writing it by hand, see reward modeling. For the same question in language models, see RLHF.

Variants

The variants are grouped by where the reward terms come from. The first group writes the terms by hand. The driving review sorts them into categories and compares ways to combine them, and the later hand-set recipes follow the same pattern.

Hand-set terms

Driving: categories and aggregation

The driving review does not compare full reward functions from end to end. It treats a reward as a set of category terms plus an aggregation rule. The first four subsections below follow that split. The last two describe whole- reward recipes from the later papers.

Safety terms

Two families are described 2405.01440 §3.1.

• Direct collision penalty. A fixed negative value when a collision occurs and zero otherwise 2405.01440 eq. 1. The review notes it ignores severity: a slow scrape and a fast impact with a pedestrian get the same penalty 2405.01440 §3.1.
• Collision penalty scaled by speed. The penalty grows with the ego vehicle's speed at impact, using a formula with a squared speed and a constant offset 2405.01440 eq. 2. The review attributes this form to earlier work 2405.01440 §3.1.
• Dense time-to-collision (TTC) term. A risk term that applies only while TTC is below a threshold t 2405.01440 eq. 3. The term f is either a constant or the inverse of TTC. The review prefers TTC to distance or headway measures, citing a comparison study 2405.01440 §3.1.

The review's own recommendation is a dense risk term plus a sparse collision penalty whose size depends on severity and on the road user hit. It is given in prose, with no equation and no values 2405.01440 §3.1.

Progress terms

The review lists these forms 2405.01440 §3.2: a delayed reward at the goal, a per-step time penalty, a dense reward for distance or speed, a penalty for deviating from a target speed, rewards for acceleration, and rewards for overtaking. Distance measured along the route is preferred to straight-line distance, because straight-line progress can reward moving away from the goal 2405.01440 §3.2.

Comfort terms

Penalties on high acceleration, deceleration, jerk and steering. The most complete definition the review found covers longitudinal and lateral acceleration and jerk, and it leaves out the rate of change of steering and of lateral acceleration. Several works omit comfort entirely [arxiv:2405.01440 §3.3].

Traffic-rule terms

Rewards for staying in lane, penalties for speeding and for undercutting a minimum headway, and rules for right-of-way and lane position when turning. Reviewed works penalise the amount of excess speed rather than enforcing speed limits. None handles several traffic laws together with relaxations 2405.01440 §3.4.

Aggregation

• Summation. Each term gets the same weight 2405.01440 eq. 4. The review says this is the most common design among the works it reviewed, and that it cannot express priorities or resolve conflicts 2405.01440 §4.1.1.
• Weighted summation. Each term has a weight 2405.01440 eq. 5. The weights are set by hand or learned with inverse reinforcement learning from expert driving. Switching weight sets by context works for simple scenes but needs much expert knowledge 2405.01440 §4.1.2. See inverse reinforcement learning.
• Lexicographic ordering. Objectives are ranked and decided in order, with no weights. Each objective needs a manual threshold, and equal priorities cannot be expressed 2405.01440 §4.1.3.

Proposed directions

These are proposals in the review. None is implemented or evaluated there.

• Rulebooks. A set of rules with a priority order, drawn as a directed graph. An action is scored by how much it violates the rules, and the priorities may be partly learned from data 2405.01440 §5.1.
• Reward machines. Finite state machines whose states carry sub-task rewards. The review suggests they could encode driving context 2405.01440 §5.2. Related: hierarchical reinforcement learning.
• Automatic reward validation. A framework that checks a reward function before training, possibly building on generation of critical scenarios. The only prior approach the review found is a manual set of eight sanity checks, which it calls hard to apply to driving 2405.01440 §5.3.

Minimal task reward for humanoid locomotion

The humanoid recipe uses a short list of terms rather than category sums. The authors state the reward has fewer than ten terms and keeps only the essential ones 2512.01996 §2.2. The terms are velocity tracking (linear and angular), swing-foot height tracking, a penalty for leaving the default pose, feet penalties (parallel feet, no crossing), an alive reward, torso uprightness penalties and an action-rate penalty 2512.01996 §2.2.

Three training choices carry part of the load. Episodes end on ground contact by the torso or by non-foot body parts 2512.01996 §2.2. Penalty weights follow a curriculum that raises them as episode length grows [arxiv:2512.01996 §2.2]. Symmetry augmentation is used 2512.01996 §2.2. The paper does not state the reward weights 2512.01996 §2.2. For whole-body motion tracking the authors follow BeyondMimic, with DeepMimic-style termination and velocity pushes 2512.01996 §2.2.

Quadruped pipe crossing: seven terms and stage training

The pipe-crossing policy uses seven reward terms 2412.13621 §II-C: linear velocity tracking with a heading term eq. 4, yaw-rate tracking eq. 5, three energy penalties on torque, torque change and joint acceleration eq. 6 eq. 7 eq. 8, a contact penalty eq. 9, and a penalty on the distance to the pipe centreline eq. 10. The fixed weights are 1.5 for linear velocity, 0.5 for yaw, 10 for contact and 0.3 for the centreline, with much smaller weights on the energy terms 2412.13621 tab. II. The paper does not compare these weights with other settings.

Training runs in three stages. Pipe radius is 0.3 to 0.5 m in stage 1, then 0.2 to 0.3 m in stages 2 and 3, and obstacles appear only in stage 3 2412.13621 tab. III. In simulation, the stage-3 policy succeeds in all 16 runs for 0.2 m and 0.3 m pipes without obstacles, and in 93.75% of runs for 0.2 m pipes with obstacles 2412.13621 tab. IV. On three real PVC pipes the robot crossed every trial without obstacles. With obstacles it reached 46.5% on one pipe 2412.13621 tab. V. The pipe-radius curriculum is a terrain curriculum that moves the robot between levels by distance travelled 2412.13621 §II-A1. It is not a reward curriculum.

Dexterous grasp: three reward groups

The reward for the in-hand stage sums three groups, with balance weights alpha, beta and eta that the paper does not state 2412.15587 eq. 6. The interaction group pulls the fingers toward the functional part of the object and the palm toward the object 2412.15587 eq. 7 eq. 8 eq. 9. The completion group rewards task progress, which the paper defines only for the laptop as its opening angle, plus a bonus that shrinks with the step count 2412.15587 eq. 10. The restriction group penalises arm motion and arm-link contact with the object 2412.15587 eq. 11 eq. 12 eq. 13. The grasp prior that sets the start pose is the main contribution, and the reward is a secondary part. In simulation the method beats DexArt on four tasks, for example 94.17% against 79.17% on the bucket 2412.15587 tab. 1. The real UR5 tests cover two tasks, with about ten trials each 2412.15587 §4.2.

Physics and task-graph shaping

Radio physics terms

For mmWave indoor navigation, the reward adds three physics terms to a distance cost: a link-state monotonicity penalty 2306.06766 eq. 12, an angle-of-arrival following penalty 2306.06766 eq. 13, and, in the weakest link state, an SNR-gradient penalty 2306.06766 eq. 14 §III-B. The paper's weights are 1.0, 0.5 and 0.3 2306.06766 tab. 2, and the global reward is grid-searched over [0.5, 2] 2306.06766 §IV-B. The paper says the shaped reward is not potential-based, so it changes the optimal policy, and it argues this helps transfer 2306.06766 §III-B.

On six unseen maps, the normalised path length in the 2+-NLOS link state is 2.46 to 2.82 for this method, against 4.50 to 5.14 for end-to-end RL 2306.06766 tab. 3. In the ablation, removing the SNR term raises the 2+-NLOS average to 4.62 and removing the link-state term raises it to 3.90, against 2.60 for the full reward 2306.06766 tab. 5. The released code's reward has neither term and feeds the policy the ground-truth link-state label, so these ablations cannot be re-run from the code. See the card for the differences.

Potential-based shaping from a reward machine

A reward machine is a finite graph of sub-goals. The reward-machine paper infers one from unlabelled RGB demonstrations 2412.10096 §4.2 §4.4. Frames are embedded by a pretrained ResNet-50, clustered with DBSCAN, and each cluster's frame closest to its mean becomes a sub-goal prototype 2412.10096 eq. 3 eq. 4 §4.3. Each prototype gives one machine state and one proposition, which is true when the feature distance is below a threshold κ 2412.10096 eq. 5 §4.4.

The shaping potential is γ raised to the number of edges from the state to the goal 2412.10096 eq. 6. The shaped transition reward adds the usual potential difference to the sparse reward 2412.10096 eq. 7 §4.4. A DQN on the inferred machine performs similarly to one given the true machine in five simulated block tasks, except Stack-3, where the baseline plateaus higher 2412.10096 §6 fig. 5. The paper gives no numeric table. The threshold κ, the clustering settings and the number of demonstrations are tuned per task, and the procedure is not described 2412.10096 tab. 1. No code is released. The driving review also lists reward machines as a future option 2405.01440 §5.2.

Learned rewards

From demonstrations: optimal transport

TemporalOT relabels the agent's trajectories with a reward from optimal transport against one or two expert videos 2410.21795 §3.1 eq. 5. Two changes make the reward order-aware: a cost averaged over k_c consecutive frames 2410.21795 eq. 6, and a band mask around the diagonal of the transport plan 2410.21795 eq. 9. The paper uses k_c = 3 and a mask half-width of 10 2410.21795 tab. 4.

Author-reported: averaged over nine Meta-World tasks with two demos, success is 61.1, against 35.9 for ADS and 20.8 for the task reward [arxiv:2410.21795 tab. 1]. The band assumes the expert moves at a similar speed. With demos sampled three times faster, Door-open success falls from 78.4 to 49.4 2410.21795 tab. 3 §3.2.3. Each reward needs a Sinkhorn solve per trajectory and per demo, about 1 h on state input and 3 h on pixels on one RTX 3090 2410.21795 app. B.4. The authors' code has no license file, so this page does not copy from it.

From a vision-language model: CLIP bonus

CLIP-RLDrive adds a bonus from an image-text model to the simulator reward. Each frame is scored against three action sentences by cosine similarity 2412.16201 eq. 12, and the bonus is added with weight W_c = 1.2 2412.16201 eq. 14 §2.4. The paper's Algorithm 1 gives a different rule: the bonus is added only when the agent's action matches the action CLIP suggests 2412.16201 alg. 1.

Author-reported on an unsignalized intersection: CLIP-DQN succeeds in 96% of episodes with one other vehicle, against 70% for DQN. With six vehicles it reaches 72%, against 50% for DQN 2412.16201 tab. 2. CLIP-PPO is below plain PPO at every vehicle count, with 38% against 78% at one vehicle 2412.16201 tab. 2. The card records that the released code gives the no-CLIP baseline the same bonus, so that comparison cannot be checked from the repository. See the card.

From a world model

Instead of writing terms, this variant reads the reward from a learned world model. A text model imagines a goal frame from the instruction. The reward then has two parts: a dense term, the LPIPS distance between the current frame and that goal frame 2512.03556 eq. 5, and a sparse term, one when the action model's predicted done signal reaches a threshold [arxiv:2512.03556 eq. 7] 2512.03556 eq. 8.

The paper is not consistent on how the two parts combine. Equation 8 adds them without weights, while Algorithm 1 mixes them with a weight alpha and 1 − alpha 2512.03556 eq. 8 2512.03556 alg. 1. The value of alpha is not stated. LPIPS is a distance, so a lower value means more similar frames. The equations add the dense term as written, and the text calls it a similarity 2512.03556 eq. 5 2512.03556 §3.1. An implementation that follows the equations literally would reward moving away from the goal, so the sign has to be fixed by the implementer.

Author-reported: policies trained with this reward reach 84.1% on unseen pick-and-place scenes with OpenVLA, against 54.5% with the simulator's rule-based reward 2512.03556 tab. 1. In-domain differences are small, and the table gives single runs with no error bars 2512.03556 tab. 1. All evaluation is in simulation 2512.03556 §5.

Redistributing a sparse team return

TAR² trains a reward model that splits one episodic team return into per-step, per-agent rewards. The temporal weights and the agent weights are non-negative, each set sums to 1, and the per-agent reward is their product times the return 2412.14779 §4.1 §4.2. The paper argues that this redistribution is potential-based shaping and so keeps the optimal policy [arxiv:2412.14779 §4.3]. The proof sets the discount to 1, while the paper's own setting uses a discount 2412.14779 §3.1 §4.3.

The learners are IPPO and MAPPO 2412.14779 §5.1. The only evaluation is one SMACLite scenario, 5m_vs_6m, shown as a figure with no numbers 2412.14779 §5.2 fig. 1. The paper's motivation is that a sparse team reward makes the advantage noisier as the number of agents grows 2412.14779 §3.4. The paper says the reward model is warmed up for 2000 episodes before use 2412.14779 §6. The released code's default is 200 episodes, and the card lists other differences.

When to use which

The driving review gives no experiments, so its rows state constraints and what the review says about each option. It does not rank the options by performance. The later rows carry the reported results of their papers. Those results are author-reported and come from different tasks, so they are not ranked against each other.

Implementation sketch

The driving review gives the aggregation forms and the safety terms as equations. The learned-reward sketch transcribes the paper's equations 5 and 7 and its Algorithm 1 combination. The other sketches transcribe the potential shaping, the CLIP bonus and the redistribution forms. The sketches below are our own transcription of those forms, not code from the papers. The papers give no numeric values for x, t, the weights, the discount or the thresholds.

# Direct collision penalty  [arxiv:2405.01440 eq. 1]
# x: negative penalty chosen by the designer; value not stated
def collision_term(collided, x):
    if collided:
        return x
    return 0.0

# Dense TTC term  [arxiv:2405.01440 eq. 3]
# t: threshold, not stated; f: constant or 1/TTC (guard TTC = 0 yourself)
def ttc_term(ttc, t, f):
    if ttc < t:
        return f(ttc)
    return 0.0

# Summation  [arxiv:2405.01440 eq. 4]
def reward_sum(r_safety, r_progress, r_comfort, r_rules):
    return r_safety + r_progress + r_comfort + r_rules

# Weighted summation  [arxiv:2405.01440 eq. 5]
# weights: hand-tuned or IRL-learned; values not stated
def reward_weighted(terms, weights):
    return sum(w * r for w, r in zip(weights, terms))

# Potential-based shaping  [arxiv:2412.10096 eq. 6] [eq. 7]
# gamma: discount, not stated in the paper; use the TD-target value
def potential_shaped(sparse, d_u, d_u_next, gamma):
    phi = gamma ** d_u
    phi_next = gamma ** d_u_next
    return sparse + gamma * phi_next - phi

# Additive CLIP bonus  [arxiv:2412.16201 eq. 12] [eq. 14]
# w: 1.2 in the paper; Algorithm 1 gates the bonus on agreement instead
def clip_bonus(r_basic, r_clip, w):
    return r_basic + w * r_clip

# Sparse done term  [arxiv:2512.03556 eq. 7]
# theta_sps: threshold, not stated
def sparse_term(done_pred, theta_sps):
    if done_pred >= theta_sps:
        return 1.0
    return 0.0

# Dense goal-distance term  [arxiv:2512.03556 eq. 5]
# lpips: lower means more similar; eq. 5 gives no sign, so decide one
def dense_term(frame, goal, lpips):
    return lpips(frame, goal)

# Learned-reward combination  [arxiv:2512.03556 alg. 1]
# alpha: weight, not stated; eq. 8 has no weights
def reward_learned(sparse, dense, alpha):
    return alpha * sparse + (1.0 - alpha) * dense

# Redistribution of an episodic return  [arxiv:2412.14779 §4.2]
# temporal_w: (T,), sums to 1; agent_w: (N, T), sums to 1 over agents
def redistribute(episode_return, temporal_w, agent_w):
    return agent_w * temporal_w * episode_return   # (N, T), sums to return

Contradictions

The papers do not test each other's claims. Where they disagree, both sides are stated and the choice is left to the implementer.

Potential-based or not. The TAR² paper argues that its redistribution is potential-based and so keeps the optimal policy 2412.14779 §4.3. The physics-terms paper states that its shaped reward is not potential-based and changes the optimal policy 2306.06766 §III-B. The reward-machine paper uses a potential-based term 2412.10096 eq. 7. The TAR² proof uses discount 1, which the paper's own setting does not use 2412.14779 §4.3.

Fixed weights. The driving review argues that fixed weighted or summed terms cannot express priorities or resolve conflicts 2405.01440 §4.1.1. The humanoid recipe uses penalty weights that change over a curriculum and reports working locomotion and tracking policies 2512.01996 §2.2 2512.01996 §3.1. The pipe-crossing paper uses fixed weights and reports stage-wise success 2412.13621 tab. II tab. IV. None of these papers compares against other weightings. The papers address different robots and tasks, so each statement stands as its authors make it.

CLIP gating. The CLIP paper's equation adds the bonus on every step, with weight 1.2 2412.16201 eq. 14. Its Algorithm 1 adds it only when the action agrees with CLIP 2412.16201 alg. 1. An implementation must pick one. The card records a third rule in the released code.

Inside the world-model paper. The text calls the LPIPS term a similarity, but LPIPS measures distance, and the equations add the term as written 2512.03556 §3.1 2512.03556 eq. 5. The paper also uses two different forms of the reward weight 2512.03556 eq. 8 [arxiv:2512.03556 alg. 1]. Implementers should pick one form and state it.

Inside the driving review. The text says speed can enter the safety penalty exponentially, but the displayed formula uses a square 2405.01440 §3.1 2405.01440 eq. 2. The review's §3 names five categories, one of them about model performance, but analyses four 2405.01440 §3. Implementers should pick one form and state it.

Paper against released code. The physics-terms paper's own reward and its released reward differ. The code has no link-state or SNR term, although the paper's ablations credit those terms 2306.06766 tab. 5. The card lists the other differences, including the weights, which differ from the paper's table.

Papers

• Review of driving reward terms: reviews reward terms for RL driving agents, compares aggregation forms, and proposes rulebooks, reward machines and reward validation without experiments.
• Humanoid sim-to-real recipe: a short humanoid reward with curriculum-ramped penalties, trained with off-policy learners on massively parallel simulation; author-reported 15-minute walking policies.
• Unified reward-observation world model: a reward read from a learned world model, a goal-frame distance plus a done signal; author-reported gains over the simulator reward on unseen scenes.
• Digital twin-enhanced wireless navigation: three radio-physics terms added to a PPO navigation reward; author-reported path-length gains on unseen maps, with the released code omitting the terms.
• Reward machine inference: infers a reward machine from unlabelled demonstrations and shapes the DQN reward with a potential; results are figures only.
• Temporal optimal transport reward: an order-aware optimal-transport reward from one or two demos; author-reported success on nine Meta-World tasks.
• Credit assignment for sparse multi-agent RL: redistributes a sparse team return into per-step, per-agent rewards; one scenario, shown only as a figure.
• Quadruped pipe crossing: a seven-term fixed-weight reward with a terrain curriculum; author-reported simulation results and few real-pipe trials.
• Dexterous manipulation with a grasp prior: a grasp-based start pose and a three-group reward for PPO; author-reported gains over DexArt in simulation.
• CLIP reward shaping for driving: adds a CLIP image-text bonus to DQN and PPO rewards at an unsignalized intersection; author-reported gains for DQN only.