There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning

In plain words

Safety testers try to trick chatbots into giving harmful answers, a practice called red teaming. Most automated testers send one clever prompt at a time and never build on what came before. Real attackers, though, probe over several messages, adjusting as the target responds.

This paper treats the attack as a multi-turn game. The attacker is rewarded for how harmful the target's answers become, and it has to plan ahead, because the payoff often comes only at the end of a conversation. The attacker is split into two parts. A planner picks a style of attack for each turn, such as posing as a journalist or a student. A writer composes the actual message, one word at a time, following that style.

Rewarding a whole message is too coarse to learn from, so the method also estimates how much each word contributed. It hides small groups of words and measures how much the target's harmful response disappears. The authors report that this attacker succeeds more often than the baselines they compared against, in their own test setup.

TL;DR

• Red teaming is modelled as a Markov decision process over dialogue turns, solved with hierarchical RL and PPO §4.1 §4.2.
• A high-level policy chooses a guide per turn; a low-level policy writes the utterance token by token §4.2.
• Each token gets a credit from masking single tokens and pairs of its top-k attended tokens, scored with a guard model §4.3 eq. 4.
• Author-reported: ASR@5 of 75.2 on HarmBench against Llama-3.1-8B-Instruct, and 35.0 against GPT-5.4-mini tab. 1.
• Code: no verified official repository; the paper links only an anonymised mirror that could not be read (see Paper vs code).

Problem

Single-turn jailbreak methods use templates or single LLM-written prompts, and prior multi-turn methods add learning but lack a sequential notion of reward §1. The authors argue that single-round setups can inflate attacker success because they deny the target the conversation history it needs to defend itself §1. Two difficulties block a sequential RL formulation: rewards are sparse and delayed until the end of a long conversation, and the space of dialogues is effectively unbounded §4 (items 1–3). The method assumes that a well-defined harm score exists for the target's responses; the authors note this may not hold for general tasks §6.

Method

Setup as an MDP. The state is the conversation history. An action is an utterance, which is a token sequence sent to the target. The transition is the target model's token distribution given the history and the utterance §4.1. Immediate reward is task-specific harm, and γ is a discount §4.1.

Hierarchy. Both levels share the same state space. The high-level action is a guide (a text string). The low-level action is a single token §4.2.

1. The high-level policy π₁ reads the full history and the target goal σ, and outputs a guide g_t §4.2.
2. The low-level policy π₂ reads the history and the guide, and generates the utterance a_t one token at a time for a fixed number k of tokens. The value of k is not stated in the text §4.2 alg. 1.
3. The target μ answers with v_t. The target is given the full history s_t, which the authors call essential for the context-aware setting §4.2.
4. The new state is s_{t+1} = s_t ‖ a_t ‖ v_t §4.2.

Harm reward. R(x₁ | x₂) is the probability that LlamaGuard assigns to the label "unsafe" for x₁ given x₂ §4.2.

High-level variants.

• Template-HL: the top 5 harmful tokens, selected by Q₁, become the "topic". A guide is sampled from a pool of 10 templates (8 personas and 2 generic follow-ups) with π₁ §4.3 app. F.1. The authors say the guidance prompts help avoid mode collapse §4.3.
• LLM-HL: an LLM writes the guide as free text, with no topic identification §4.3 app. F.2.

High-level objective. The high-level critic Q₁ is defined by a one-step target with the immediate reward R(v_t | s_t ‖ a_t) − R(a_t | s_t), plus a discounted value of the next state eq. 1. The high-level policy is trained with PPO to maximise Q₁ §4.3.

Q_1(s_t, g_t, \sigma) = \mathbb{E}\Big[ R(v_t \mid s_t \,\|\, a_t) - R(a_t \mid s_t) + \gamma V_1(s_{t+1}, \sigma) \Big] eq. 1

where the expectation is over g_t ~ π₁, a_t ~ π₂ and v_t ~ μ, and V₁(s_{t+1}, σ) is the expectation of Q₁ over the next guide eq. 1 §4.3.

Turn reward for the low level. The turn score G adds to the harm gap a cosine similarity J between reference-LLM embeddings of the guide and the utterance. The similarity term keeps the low-level policy on the guide and stops it from overfitting one utterance §4.3 eq. 2.

\mathcal{G}(s_t, g_t, a_t, s_{t+1}) = R(v_t \mid s_t \,\|\, a_t) - R(a_t \mid s_t) + J(g_t, a_t) eq. 2

J(g_t, a_t) = \frac{\omega_{g_t} \cdot \omega_{a_t}}{\lVert \omega_{g_t} \rVert \, \lVert \omega_{a_t} \rVert} eq. 2

ω is the embedding of an input from a reference LLM §4.3.

Token credit (naive marginal). The reward for token τ_i is the drop in G when that token is removed from the utterance. The removed utterance is scored with the same response s_{t+1} §4.3 eq. 3.

r_2(\tau_i) = \mathcal{G}(s_t, g_t, a_t, s_{t+1}) - \mathcal{G}(s_t, g_t, a_t - \mathrm{seq}(\tau_i), s_{t+1}) eq. 3

Token credit (with interactions). Single-token masks ignore cases where harm needs two tokens together. The method therefore masks subsets of size 1 or 2 drawn from the top-k attended tokens of the utterance, as scored by the guard model §4.3. Let M_τ be the set of masks that contain τ_i. The reward averages the masked drops over M_τ eq. 4.

r_2(\tau_i) = \frac{1}{|\mathcal{M}_{\tau_i}|} \sum_{m \in \mathcal{M}_{\tau_i}} \Big[ \mathcal{G}(s_t, g_t, a_t, s_{t+1}) - \mathcal{G}(s_t, g_t, a_t - m, s_{t+1}) \Big] eq. 4

Low-level value. The low-level critic V₂ is a Bellman backup over the next-token distribution of π₂ eq. 5 §4.3.

V_2(\tau_i) = r_2(\tau_i) + \gamma \sum_{\tau_{i+1}} P_{\pi_2}(\tau_{i+1}) \, V_2(\tau_{i+1}) eq. 5

Training. Both levels are trained with PPO §4.3 app. D.5. The low-level agent generates several candidate tokens. It takes the argmax of Q₁ with probability 1 − ε, a form of rejection sampling that the authors use to improve exploration §4.3.

Algorithm 1 (Hierarchical PPO). The algorithm gives the training loop in order alg. 1 app. D.5.

# Hierarchical PPO rollout and update  [alg. 1]
# Input: target goal sigma, target model mu, guard R
# Output: updated high-level policy theta, low-level policy phi

def train_episode(sigma, target_model, guard):
    # Start from an empty history  [alg. 1]
    history = ""
    # Number of turns per episode is not stated  [app. E]
    for step in range(num_turns):
        # High level: pick a guide from the history and goal  [alg. 1] [§4.3]
        guide = high_level_policy(history, sigma)
        # Low level: write the utterance token by token  [alg. 1] [§4.2]
        utterance = low_level_policy(history, guide)
        # Target answers with the full history  [alg. 1] [§4.2]
        response = target_model(history + utterance)
        # Fit Q1 on the target value of the step  [alg. 1] [eq. 1]
        fit_high_level_critic(history, utterance, sigma, response)
        # Fit V2 on each token of the utterance  [alg. 1] [eq. 5]
        for token in utterance:
            fit_low_level_critic(token, history, guide, utterance)
        # Update the low-level policy to maximise V2  [alg. 1]
        update_low_level_policy()
        history = history + utterance + response
    # Update the high-level policy to maximise Q1  [alg. 1]
    update_high_level_policy()

Note the Q₁ update in Algorithm 1 is written as Q_{1,ψ}(s_t, a_t, σ), without g_t, while Eq. 1 conditions on g_t. See Pitfalls.

Token-credit computation. The following sketch implements eq. 2 to eq. 4 with the masks of size 1 and 2. Masked utterances are scored against the original response eq. 3 eq. 4.

# Per-token credit from masked utterances  [eq. 2] [eq. 4]
# Masks keep positions: a masked token becomes null  [§3]
def token_rewards(history, guide, utterance, response, top_k):
    # Full score G for the unmasked utterance  [eq. 2]
    full_score = turn_score(history, guide, utterance, response)
    # Only the top-k attended tokens can appear in a mask  [§4.3]
    key_positions = top_attended_positions(utterance, top_k)
    rewards = []
    for position in range(len(utterance)):
        # Masks of size 1 or 2 that contain this position  [eq. 4]
        masks = masks_containing(position, key_positions)
        # |masks| is 0 for tokens outside the top-k: not defined in paper
        total = 0.0
        for mask in masks:
            masked = apply_mask(utterance, mask)
            # The response stays fixed, as in s_{t+1} of eq. 3  [eq. 3]
            total += full_score - turn_score(history, guide, masked, response)
        rewards.append(total / len(masks))
    return rewards

Cost. The authors say the credit step multiplies the cost of plain reward-driven fine-tuning by n·k, where k is the number of interacting tokens (k = 5 in the experiments). They also report a negligible runtime increase, because the guard runs one forward pass per candidate rather than generating a sequence app. C. At test time the attacker is an ordinary autoregressive model app. C.

Hyperparameters

Setting of the evaluation: ASR@n counts an episode as a success if the target's response at turn ≤ n is toxic §5. Myopic targets see only the latest attacker message; context-aware targets see the whole dialogue. In both, the attacker sees the whole dialogue §5.

Results

Author-reported. Setup: attacks are trained against an 8B open-source target on HarmBench data, then evaluated on HarmBench, JailbreakBench and WildBench; baselines were reproduced by the authors in one shared setting §5. The main headline table (few-shot, ASR@5) is below tab. 1.

Table 1 (HarmBench, ASR@5). TtS is mean turns to success; "-" means no successes.

tab. 1

Table 2 (ASR@30, context-aware, 30-step environment). Columns give ASR on HarmBench (HB), WildBench (WB) and JailbreakBench (JB) per target; the last columns average PPL (perplexity, lower is better), TtS (turns to success) and Div (pairwise cosine distance of attack embeddings, higher is more diverse). The table also contains two ablation rows from the paper.

tab. 2

Table 3 (sensitivity and ablation, HarmBench ASR@5, Llama-3.1-8B target).

tab. 3

Table 4 (appendix, ASR@30, adds Mixtral-8x22b and GCG and Rainbow-Teaming). Column groups are HB, WB and JB per target, plus GPT-4o HB only. "--" means not reported.

tab. 4

Headline claims as the paper states them. The method "outperforms baselines" on HarmBench few-shot ASR@5 (tab. 1) and "maintains SOTA" in the ASR@30 setting (tab. 2) §5.1. The authors report larger gains in the context-aware setting than in the myopic one §5.1. The abstract calls this a new state of the art [abstract]. Table 1 shows the method is not best on every column: WildTeaming has the lowest TtS on Llama-8B myopic (3.53), and X-Teaming has the lowest TtS on GPT-5.4-mini context-aware (4.0) tab. 1. For diversity, LLM-HL has the highest Div in tab. 2 (0.92; the best baseline is 0.72). For perplexity, the authors say LLM-HL is low §5.1, but its average PPL (13.3) is higher than Template-HL (7.2), X-Teaming (7.2) and MTSA (6.9) in the same table.

Ablation findings. Freezing either level lowers ASR@5 from 75.2 to 40.7 (low frozen) or 47.0 (high frozen), and removing the hierarchy gives 31.2 tab. 3. Stronger guards in training correlate with a stronger attacker tab. 3a §5.2. Compared with uniform and exponential-decay token credit, and with the no-interaction variant, the pairwise marginal credit gives the best training curves §5.2 fig. 3 app. B.

Paper vs code

No official code. The paper's only code link is an anonymised mirror, https://anonymous.4open.science/r/rlrt-5843 app. A. Anonymised mirrors do not establish that the repository belongs to the authors, and the mirror returned only its title page when read, so I could not verify a LICENSE file, a commit hash or the training code. No code, code_commit or code_license is recorded for this reason. Searches done: the arXiv abstract page (no code, project or DOI link beyond the DOI), GitHub repository search for rlrt red teaming hierarchical and hierarchical red teaming reinforcement (no results). No project page was found.

Pitfalls

• Base model conflicts. The experiments section names Llama-3-8B-Instruct as the base model §5, but the architecture appendix and all result tables use Llama-3.1-8B-Instruct app. D.1 tab. 1. Use Llama-3.1-8B-Instruct, the model actually evaluated, and treat the §5 wording as a typo.
• Two different k values. The template variant takes the top 5 harmful tokens as the topic §4.3. The credit step takes k = 5 attended tokens for masks app. C. The text does not say whether these are the same parameter.
• Masked utterances reuse the response. Eq. 3 and eq. 4 keep s_{t+1} fixed when scoring masked utterances eq. 3 eq. 4. The paper does not say whether the target was re-queried. Re-querying would change the cost and the results, so keep the response fixed unless a later source says otherwise.
• Tokens outside the top k get no mask. Eq. 4 divides by |M_τ|, which is zero for any token that is not among the top-k attended tokens. The paper does not state the reward for such tokens, and the pseudocode above marks this case as undefined.
• Q₁ conditioning mismatch. Eq. 1 conditions Q₁ on the guide g_t. Algorithm 1 writes the Q₁ update as Q_{1,ψ}(s_t, a_t, σ), without g_t alg. 1. Condition on the guide, as the equation does, unless the code shows otherwise.
• Equation 5 needs the full next-token distribution. V₂ is a sum over every next token weighted by π₂ eq. 5. Computing it exactly over the vocabulary is expensive. The paper does not say whether it was approximated, and Algorithm 1 only states "update η to fit V₂".
• Reward magnitudes are implied. R is a probability in [0, 1] and J is a cosine in [−1, 1], so G is bounded by [−2, 2]. The paper gives no weights for the three terms, so each has weight 1 by default.
• Rejection sampling is under-specified. ε and the number of candidate tokens are not stated §4.3.
• Evaluation judge is not stated. ASR depends on "a judge" §5. The training reward is LlamaGuard, but the text does not say which judge produced the reported ASR values.
• Baselines were reproduced by the authors. They say baselines "must be reproduced in a novel setting" §5. These numbers are not the original papers' numbers, and they are measured under the authors' context and step settings.
• Tables 2 and 4 disagree on some cells. Values that differ between tab. 2 and tab. 4 for the same method and target: LLM-HL Llama-8B HB (95.0 vs 97.0); Template-HL Llama-8B HB (96.0 vs 97.0); LLM-HL Llama-70B HB (86.5 vs 87.0); X-Teaming Llama-70B HB (22.1 vs 12.1); MTSA Llama-8B HB (77.0 vs 87.0). The text does not explain the gap. Use tab. 2 for the main comparison, and mark tab. 4 as inconsistent.
• Low-level-only results differ between tables. The ablation in tab. 3b reports 31.2 ASR@5 for no-HRL, while tab. 2 reports 35.0 ASR@30 for the low level alone. These are different settings and should not be merged.
• Cost claim. The n·k cost increase app. C refers to the reward computation. Training also runs 5 epochs over each batch and needs a second GPU for target and guard app. E.
• Context dependence. The headline gains are largest in the context-aware setting, where the target sees the full history §5.1. The method's advantage over myopic baselines therefore depends on that setting.
• Harmful examples. Appendix G contains harmful sample dialogues app. G. They are not reproduced here and should not be copied into prompts or datasets.
• Equation numbers are inferred. Numbers were assigned from the LaTeX source in order of numbered align environments eq. 1 to eq. 5. Check against the compiled PDF before citing externally.
• Figure references. Figures are described rather than included. fig. 1 is the example dialogue; fig. 2 is the HRL overview; fig. 3 has two panels, (a) the adversarial reward ablation and (b) the reward attribution comparison; fig. 4 is the strategy transition heatmap; fig. 5 has four loss curves for the two critics and two policies §4 §5 app. B app. D.

Lineage

• Builds on: PPO §4.3; LoRA §5 app. D; hierarchical RL for sparse rewards §2; LlamaGuard as the harm reward §4.2. The paper cites ArCHer as the closest prior hierarchical language-RL method and states its difference from it §2. None of these papers has a card in this wiki.
• Compared against: Ferret, WildTeaming, HARM, PAIR, X-Teaming, MTSA, APRT tab. 1 tab. 2; Rainbow-Teaming and GCG in the appendix tab. 4.
• Superseded by: not stated.