---
kind: concept
slug: constrained-reinforcement-learning
title: Constrained reinforcement learning
summary: >-
  Maximise expected return while keeping an expected cost under a limit, usually by a Lagrange multiplier that rises while the cost is over budget. One card so far: HCRide, a multi-agent ride-hailing dispatcher, with author-reported gains over CPO and Lag-TRPO on its own simulator. Its released code does not apply the cost term to the actor.
tags: [reinforcement-learning, constraints, safety]
papers: ["2508.04811"]
related: [proximal-policy-optimization, actor-critic, policy-gradient, multi-agent-reinforcement-learning, reward-design, safety-filter, algorithmic-fairness]
status: draft
updated_at: 2026-11-05
---

## Overview

A constrained Markov decision process keeps the usual reward and adds a cost
signal with a limit. The learner maximises the expected discounted reward J_r
while keeping the expected discounted cost J_c at or below a limit ξ
[arxiv:2508.04811 §3.1] [arxiv:2508.04811 eq. 9].

The standard handle is a Lagrangian, L(θ, λ) = J_r − λ(J_c − ξ), with λ ≥ 0.
The policy ascends L in θ, while λ is raised when the cost is over its limit
and lowered when it is under [arxiv:2508.04811 eq. 10] [arxiv:2508.04811 §3.2].
The two updates alternate until both settle [arxiv:2508.04811 eq. 11]–[arxiv:2508.04811 eq. 13].

This page rests on one card. The other methods the card compares against
(CPO, Lag-TRPO and the multi-agent baselines) have no card here, so they are
named below but not described.

## Variants

### Lagrangian actor with reward and cost critics (Habic)

The card's method, Habic, treats ride-hailing dispatch as a constrained MDP.
Each driver is an agent. Training is centralised and execution is
decentralised [arxiv:2508.04811 §3.1]. The reward mixes a wait-time term with
a fairness term weighted by α [arxiv:2508.04811 eq. 6]. The cost is incurred
when a dispatch ends in a region the driver does not prefer, with a limit of
15% in the study [arxiv:2508.04811 §4.2.3].

The policy is one shared network that scores a short list of candidate drivers
for each order, and a softmax over the scores gives the choice probabilities
[arxiv:2508.04811 eq. 14] [arxiv:2508.04811 §3.3.2]. It is trained with a
PPO-clip objective whose direction for θ is the reward gradient minus λ times
the cost gradient [arxiv:2508.04811 eq. 15]. Two critics estimate reward and
cost separately, and λ takes a projected step [arxiv:2508.04811 §3.3.3]
[arxiv:2508.04811 eq. 16]. Card: [HCRide](/papers/2508.04811).

Reported results are the authors' own, on a simulator built from two
ride-hailing datasets [arxiv:2508.04811 §4.1]. On the NYC setting, the
introduction gives gains over Lag-TRPO of 2.02% on efficiency and 10.21% on
driver preference [arxiv:2508.04811 §1]. The full table includes CPO and
Lag-TRPO as baselines. The method is bold on every NYC metric, and on every
Shenzhen metric except driver preference, where a sibling variant is higher
[arxiv:2508.04811 tab. 1]. The table's caption conflicts with its values (see
Contradictions).

### Other variants

No second card lists this concept yet. Methods named as baselines in the card
(CPO, Lag-TRPO) are not described on this page.

## When to use which

| Situation | Use | Why |
|---|---|---|
| One cumulative cost with a fixed limit, and a policy-gradient learner | Lagrangian form: reward minus λ times the cost gap, λ kept at zero or above | This is the formulation the card states and uses [arxiv:2508.04811 §3.2] [arxiv:2508.04811 eq. 10]. The card does not test it against other constraint handlers. |
| Many agents share one policy and each decision has a small candidate set | Shared actor that scores candidates, with separate reward and cost critics | This is the card's setting and design [arxiv:2508.04811 §3.3]. Evidence covers one simulator family only [arxiv:2508.04811 §4.1]. |
| You need the cost to change the learned policy | Check the update in code before reuse | The card's paper-vs-code review records that the released actor loss has no cost term, so the cost cannot change the policy there [see card: Paper vs code, item 5]. |
| You need results you can cite as independent | Treat the numbers as author-reported | All results come from the authors' own simulator. The number of runs is not stated and the "±" spread is not defined [arxiv:2508.04811 tab. 1]. |

## Implementation sketch

One outer iteration of the Lagrangian update. Steps follow the card's
equations; the multiplier step uses the sign that raises λ when the cost
estimate exceeds the limit.

```pseudocode
# One outer iteration of a Lagrangian constrained policy update  [§3.2]
# theta: policy parameters; lam: multiplier kept >= 0; xi: cost limit
def constrained_update(theta, lam, batch, xi, lr_theta, lr_lam):
    # Estimate reward return and cost return on the same batch  [§3.3.3]
    J_r = estimate_reward_return(batch)
    J_c = estimate_cost_return(batch)
    # Ascend the Lagrangian in theta: reward gradient minus lam times cost  [eq. 15]
    grad = grad_reward(theta, batch) - lam * grad_cost(theta, batch)
    theta = theta + lr_theta * grad
    # Projected step on lam: raise it while the cost is over the limit  [eq. 16]
    lam = max(0.0, lam + lr_lam * (J_c - xi))
    return theta, lam
```

## Contradictions

- **Paper vs its released code.** The paper's θ update includes the cost
  gradient [arxiv:2508.04811 eq. 15]. The card records that the released actor
  loss uses only the reward surrogate, and that the code's constraint quantity
  is an advantage-based estimate rather than the cost return in eq. 9
  [see card: Paper vs code, items 5–6]. The two sides are not reconciled here.
- **Table 1 caption vs its values.** The caption says lower values are better,
  while the bold marks and the text treat higher as better
  [arxiv:2508.04811 tab. 1] [arxiv:2508.04811 §4.2.3]. Read each column by its
  own numbers.
- **Discount factors.** The paper defines separate γ_r and γ_c
  [arxiv:2508.04811 §3.1]. The released code uses one value for both
  [see card: Paper vs code, item 9].

## Papers

- [HCRide](/papers/2508.04811): a constrained-MDP ride-hailing dispatcher (Habic) with a driver-preference cost limit; author-reported gains on its own simulator.