Jailbreak attacks

Overview

A jailbreak attack is an input, often optimised by search, that gets a safety-tuned model to give a reply it was trained to refuse. The attacker needs two things: an objective that the search can push on, and a judge that decides whether a reply counts as a success 2412.10321 §1 2412.10321 §2.

The common family in the source card is prefix forcing. The attacker fixes the start of the reply, for example an affirmative opening, and optimises the attack prompt so that the model is likely to begin with it. The card treats this as a proxy. Lowering the loss on the prefix does not reliably produce a reply that is useful to the attacker 2412.10321 §3.2. A reply only counts as a nuanced jailbreak if it is affirmative, complete and faithful to the request 2412.10321 §2.

Every number on this page comes from the authors' own experiments and their own judges. Other judges give different success rates on the same outputs, see Contradictions.

Variants

Fixed-prefix objective

The attack prompt is optimised to minimise the negative log-likelihood of one fixed target prefix after the request 2412.10321 eq. 2 2412.10321 §3.1. The authors argue two faults with this target 2412.10321 §3.2:

• Misspecified. Prefilling the prefix removes direct refusals, but the continuation is often incomplete or unfaithful, so success stays low.
• Overconstrained. A fixed wording can be unnatural for a given model. On Llama-3 the authors observe that "Here" is more common than the fixed target, and a different prefix gave lower final losses 2412.10321 §3.2 app. D.

AdvPrefix: model-dependent multi-prefix objective

The objective replaces the single target with a set of prefixes and minimises the negative log of their summed probability 2412.10321 eq. 3 2412.10321 §4.1. Any member of the set can count as success, which relaxes the fixed-wording constraint. The set is chosen per victim model and per request, using two criteria 2412.10321 §4.2:

• a high prefilling success rate, estimated with a hand-built attack prompt and judged by the nuanced judge 2412.10321 eq. 4 2412.10321 §4.2;
• a low initial negative log-likelihood under the victim's chat context 2412.10321 eq. 5.

The two criteria pull against each other, because longer prefixes can succeed more often but cost more to produce. The authors combine them as a weighted sum of log success and NLL 2412.10321 §4.2. The paper's pipeline generates candidates with uncensored helper models, and can steer their decoding with probabilities from the victim 2412.10321 §4.3.

The objective plugs into existing attacks. The card reports runs with GCG, optimising a suffix or the whole prompt, and with AutoDAN 2412.10321 §5. GCG and AutoDAN are not covered by a card in this repository, so they are named here only. The paper's AdvPrefix card lists them as reference points.

Single prefix versus several

The paper reports results with one prefix and with four 2412.10321 §5 2412.10321 tab. 1. Four prefixes did not beat one consistently in the suffix-only table. For Llama-3 at 20 tokens, one prefix reached 54.6% and four reached 54.0%. For Llama-3.1 at 20 tokens, four reached 60.0% against 45.0% for one 2412.10321 tab. 1.

Judges as a variant

The same outputs can be scored by different judges. The paper's nuanced judge is Llama-3.1-70B, instructed to score affirmative, complete and faithful replies 2412.10321 §2. The paper also uses a pairwise preference judge that compares a reply with an uncensored model's reply 2412.10321 app. C 2412.10321 §5, and compares its results with HarmBench, JailbreakBench and StrongReject judges 2412.10321 tab. 3.

When to use which

Cost: the card reports about five minutes per prefix on one A100 80GB, and about three minutes with a rougher success estimate. The 70B judge dominates the cost 2412.10321 §5.

Implementation sketch

Our own pseudocode for the multi-prefix loss 2412.10321 eq. 3. A forward pass per prefix is the simplest form. The paper notes that several prefixes can be batched with tree attention 2412.10321 §4.1.

# Multi-prefix loss over an attack prompt theta  [arxiv:2412.10321 eq. 3]
# Input:  request x, prefix set Yp, attack prompt theta
# Output: scalar loss to minimise

def multi_prefix_loss(theta, x, prefixes):
    # Log-probability of each prefix after the request and attack prompt
    logps = [log_prob(prefix, x, theta) for prefix in prefixes]
    # Negative log of the summed probability over the set  [eq. 3]
    return -logsumexp(logps)

Prefix scoring, run per victim model and request 2412.10321 §4.2:

# Score one candidate prefix  [arxiv:2412.10321 §4.2] [arxiv:2412.10321 §5]
# Input:  victim, request, candidate, judge, n_samples
# Output: estimated prefilling success rate (PASR)

def estimate_pasr(victim, x, prefix, judge, n_samples):
    hits = 0
    for _ in range(n_samples):
        # Force the prefix, then sample the rest at temperature 1 [§5]
        rest = victim.sample_after(x, prefix, temperature=1.0)
        reply = prefix + rest
        # Replies below a short length cutoff count as failures [§5]
        if len(reply) < 300:
            continue
        if judge.is_jailbreak(x, reply):
            hits += 1
    return hits / n_samples

Notes for implementers:

• The paper's values are in the Hyperparameters table of the card. Where the paper says "not stated", do not invent a value.
• The 300-character cutoff on replies is a detail of the released code. The paper text does not mention it, so a reimplementation should decide whether to keep it and say so in its results.
• The paper gives 20 as the weight on log success 2412.10321 §5, while the released code defaults to 10. Choose one and report it.

Contradictions

• Text versus table, full-prompt results. The text gives Llama-2 as 39% to 73% and Llama-3 as 70% to 80% under full-prompt optimisation 2412.10321 §5. Table 2 gives GCG as 42.1 to 72.6 for Llama-2 and 14.1 to 79.5 for Llama-3 2412.10321 tab. 2. Both come from the same paper, and the ranges differ.
• Headline depends on setting. The introduction's 16% to 70% for Llama-3 matches the 40-token suffix case. The 20-token case is 12.8% to 54.6% 2412.10321 §1 2412.10321 tab. 1. The setting must be stated with any headline.
• Judges disagree on the same outputs. For the Gemma-2 full-prompt GCG outputs under the new objective, the nuanced judge gives 51.2%. HarmBench, JailbreakBench and StrongReject give 66.3%, 70.6% and 64.3% 2412.10321 tab. 3. The authors' meta-evaluation against 800 labelled replies rates the nuanced judge highest on average, with agreement 91.8% and F1 85.4 2412.10321 tab. 4. Those labels are the authors' own.
• Overestimation by existing judges. The authors report that existing judges overestimate nuanced success on the original objective, by up to 30 points 2412.10321 §1 2412.10321 fig. 2.

Papers

• AdvPrefix: An Objective for Nuanced LLM Jailbreaks: replaces the fixed prefix target with model-dependent prefix sets chosen for prefilling success and low NLL, and plugs them into GCG and AutoDAN. Author-reported gains on Llama-2, Llama-3, Llama-3.1 and Gemma-2, scored by the authors' nuanced judge.