---
kind: concept
slug: jailbreak-attacks
title: Jailbreak attacks
summary: >-
  Adversarial prompts that get a safety-tuned chat model to produce disallowed replies. Covers the prefix-forcing objective ("start with an affirmative phrase") and its model-dependent replacement, AdvPrefix, used with GCG or AutoDAN. Success is set by a judge, so numbers depend on which judge is used. Results are author-reported.
tags: [safety, jailbreak, red-teaming, prompt-optimization, llm-safety]
papers: ["2412.10321"]
related: [red-teaming, llm-as-a-judge, adversarial-text-generation, adversarial-robustness, safety-filter, llm-ethics]
status: draft
updated_at: 2026-11-08
---

## Overview

A jailbreak attack is an input, often optimised by search, that gets a
safety-tuned model to give a reply it was trained to refuse. The attacker needs
two things: an objective that the search can push on, and a judge that decides
whether a reply counts as a success [arxiv:2412.10321 §1] [arxiv:2412.10321 §2].

The common family in the source card is prefix forcing. The attacker fixes the
start of the reply, for example an affirmative opening, and optimises the
attack prompt so that the model is likely to begin with it. The card treats this
as a proxy. Lowering the loss on the prefix does not reliably produce a reply
that is useful to the attacker [arxiv:2412.10321 §3.2]. A reply only counts as a
nuanced jailbreak if it is affirmative, complete and faithful to the request
[arxiv:2412.10321 §2].

Every number on this page comes from the authors' own experiments and their own
judges. Other judges give different success rates on the same outputs, see
[Contradictions](#contradictions).

## Variants

### Fixed-prefix objective

The attack prompt is optimised to minimise the negative log-likelihood of one
fixed target prefix after the request [arxiv:2412.10321 eq. 2] [arxiv:2412.10321 §3.1].
The authors argue two faults with this target [arxiv:2412.10321 §3.2]:

- **Misspecified.** Prefilling the prefix removes direct refusals, but the
  continuation is often incomplete or unfaithful, so success stays low.
- **Overconstrained.** A fixed wording can be unnatural for a given model. On
  Llama-3 the authors observe that "Here" is more common than the fixed target,
  and a different prefix gave lower final losses [arxiv:2412.10321 §3.2] [app. D].

### AdvPrefix: model-dependent multi-prefix objective

The objective replaces the single target with a set of prefixes and minimises
the negative log of their summed probability [arxiv:2412.10321 eq. 3]
[arxiv:2412.10321 §4.1]. Any member of the set can count as success, which
relaxes the fixed-wording constraint. The set is chosen per victim model and per
request, using two criteria [arxiv:2412.10321 §4.2]:

- a high prefilling success rate, estimated with a hand-built attack prompt and
  judged by the nuanced judge [arxiv:2412.10321 eq. 4] [arxiv:2412.10321 §4.2];
- a low initial negative log-likelihood under the victim's chat context
  [arxiv:2412.10321 eq. 5].

The two criteria pull against each other, because longer prefixes can succeed
more often but cost more to produce. The authors combine them as a weighted sum
of log success and NLL [arxiv:2412.10321 §4.2]. The paper's pipeline generates
candidates with uncensored helper models, and can steer their decoding with
probabilities from the victim [arxiv:2412.10321 §4.3].

The objective plugs into existing attacks. The card reports runs with GCG,
optimising a suffix or the whole prompt, and with AutoDAN [arxiv:2412.10321 §5].
GCG and AutoDAN are not covered by a card in this repository, so they are named
here only. The paper's [AdvPrefix card](/papers/2412.10321) lists them as
reference points.

### Single prefix versus several

The paper reports results with one prefix and with four [arxiv:2412.10321 §5]
[arxiv:2412.10321 tab. 1]. Four prefixes did not beat one consistently in the
suffix-only table. For Llama-3 at 20 tokens, one prefix reached 54.6% and four
reached 54.0%. For Llama-3.1 at 20 tokens, four reached 60.0% against 45.0%
for one [arxiv:2412.10321 tab. 1].

### Judges as a variant

The same outputs can be scored by different judges. The paper's nuanced judge is
Llama-3.1-70B, instructed to score affirmative, complete and faithful replies
[arxiv:2412.10321 §2]. The paper also uses a pairwise preference judge that
compares a reply with an uncensored model's reply [arxiv:2412.10321 app. C]
[arxiv:2412.10321 §5], and compares its results with HarmBench, JailbreakBench
and StrongReject judges [arxiv:2412.10321 tab. 3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Attacking a model with a full-prompt optimiser (GCG, 40 tokens) on Llama-3 | Fixed-prefix objective as the baseline; AdvPrefix objective for the comparison | The paper reports 14.1% against 79.5% nuanced success [arxiv:2412.10321 tab. 2] |
| Suffix-only optimisation on Llama-3.1 | Either variant; test first | Four prefixes reached 60.0% against 16.8% baseline at 20 tokens [arxiv:2412.10321 tab. 1]. Full-prompt GCG on Llama-3.1 moved only from 47.0% to 58.9% [arxiv:2412.10321 tab. 2] |
| Victim is a reasoning model | AdvPrefix, with model-dependent prefixes | On an R1-distilled Llama-3-8B the authors report that default prefixes could not lower the loss enough, and model-dependent prefixes let the attack work [arxiv:2412.10321 fig. 9] |
| No access to the victim's token probabilities | Not covered | The objective is computed from the victim's log-probabilities [arxiv:2412.10321 eq. 3]. The card does not study black-box use |
| No budget for a 70B judge | Not established by the card | Prefix selection and scoring both use the nuanced judge [arxiv:2412.10321 §5]. The card does not test a smaller judge in selection |
| Need results comparable with other papers | Report several judges | Judge choice changes the numbers a lot (see Contradictions) |

Cost: the card reports about five minutes per prefix on one A100 80GB, and
about three minutes with a rougher success estimate. The 70B judge dominates the
cost [arxiv:2412.10321 §5].

## Implementation sketch

Our own pseudocode for the multi-prefix loss [arxiv:2412.10321 eq. 3]. A
forward pass per prefix is the simplest form. The paper notes that several
prefixes can be batched with tree attention [arxiv:2412.10321 §4.1].

```pseudocode
# Multi-prefix loss over an attack prompt theta  [arxiv:2412.10321 eq. 3]
# Input:  request x, prefix set Yp, attack prompt theta
# Output: scalar loss to minimise

def multi_prefix_loss(theta, x, prefixes):
    # Log-probability of each prefix after the request and attack prompt
    logps = [log_prob(prefix, x, theta) for prefix in prefixes]
    # Negative log of the summed probability over the set  [eq. 3]
    return -logsumexp(logps)
```

Prefix scoring, run per victim model and request [arxiv:2412.10321 §4.2]:

```pseudocode
# Score one candidate prefix  [arxiv:2412.10321 §4.2] [arxiv:2412.10321 §5]
# Input:  victim, request, candidate, judge, n_samples
# Output: estimated prefilling success rate (PASR)

def estimate_pasr(victim, x, prefix, judge, n_samples):
    hits = 0
    for _ in range(n_samples):
        # Force the prefix, then sample the rest at temperature 1 [§5]
        rest = victim.sample_after(x, prefix, temperature=1.0)
        reply = prefix + rest
        # Replies below a short length cutoff count as failures [§5]
        if len(reply) < 300:
            continue
        if judge.is_jailbreak(x, reply):
            hits += 1
    return hits / n_samples
```

Notes for implementers:

- The paper's values are in the Hyperparameters table of the card. Where the
  paper says "not stated", do not invent a value.
- The 300-character cutoff on replies is a detail of the released code. The
  paper text does not mention it, so a reimplementation should decide whether to
  keep it and say so in its results.
- The paper gives 20 as the weight on log success [arxiv:2412.10321 §5], while
  the released code defaults to 10. Choose one and report it.

## Contradictions

- **Text versus table, full-prompt results.** The text gives Llama-2 as 39% to
  73% and Llama-3 as 70% to 80% under full-prompt optimisation [arxiv:2412.10321 §5].
  Table 2 gives GCG as 42.1 to 72.6 for Llama-2 and 14.1 to 79.5 for Llama-3
  [arxiv:2412.10321 tab. 2]. Both come from the same paper, and the ranges differ.
- **Headline depends on setting.** The introduction's 16% to 70% for Llama-3
  matches the 40-token suffix case. The 20-token case is 12.8% to 54.6%
  [arxiv:2412.10321 §1] [arxiv:2412.10321 tab. 1]. The setting must be stated
  with any headline.
- **Judges disagree on the same outputs.** For the Gemma-2 full-prompt GCG
  outputs under the new objective, the nuanced judge gives 51.2%. HarmBench,
  JailbreakBench and StrongReject give 66.3%, 70.6% and 64.3%
  [arxiv:2412.10321 tab. 3]. The authors' meta-evaluation against 800 labelled
  replies rates the nuanced judge highest on average, with agreement 91.8% and F1
  85.4 [arxiv:2412.10321 tab. 4]. Those labels are the authors' own.
- **Overestimation by existing judges.** The authors report that existing judges
  overestimate nuanced success on the original objective, by up to 30 points
  [arxiv:2412.10321 §1] [arxiv:2412.10321 fig. 2].

## Papers

- [AdvPrefix: An Objective for Nuanced LLM Jailbreaks](/papers/2412.10321): replaces the fixed prefix target with model-dependent prefix sets chosen for prefilling success and low NLL, and plugs them into GCG and AutoDAN. Author-reported gains on Llama-2, Llama-3, Llama-3.1 and Gemma-2, scored by the authors' nuanced judge.