AdvPrefix: An Objective for Nuanced LLM Jailbreaks

In plain words

Some ways of getting a chatbot past its safety rules work by steering the start of its reply. The attacker picks a fixed opening such as "Sure, here is ...", then searches for an input that makes the model begin with that opening. The authors argue that one fixed opening is a poor target. Models often say it and then give a vague, partial or unrealistic answer, and some models rarely start with that wording at all.

AdvPrefix replaces the single fixed opening with one or more openings chosen for each model and each request. A good opening is one that, once the model has started it, tends to lead to a complete and realistic answer, and that the model finds easy to produce. Candidate openings come from helper models that have had their refusals removed, and a judge checks how often each one leads to a harmful answer. The authors report that this raises measured success considerably on several open models. Those numbers come from their own judge and their own tests.

TL;DR

Prefix-forcing jailbreaks make a model begin with one fixed phrase, such as "Sure, here is ...", and optimise the attack input to make that phrase likely. The authors find that this phrase is a poor target: models often start it and then give incomplete or unfaithful replies, and the phrase is unnatural for some models §3.2. AdvPrefix instead targets one or more prefixes per model and per request, chosen for high prefilling attack success and low initial NLL §4.2, and plugs them into GCG or AutoDAN §5. The reported gain is large, for example Llama-3 full-prompt GCG rises from 14.1% to 79.5% nuanced ASR tab. 2. The judge that scores these results is the authors' own 70B model §2.

Problem

The common objective minimises the negative log-likelihood of one fixed prefix y_p given the attack prompt \theta eq. 2 §3.1. Two limitations follow §3.2:

• Misspecified. Lowering this loss does not reliably raise nuanced success. Prefilling the prefix removes direct refusals, but incomplete and unfaithful continuations remain, so success stays low §3.2 fig. 3.
• Overconstrained. A rigid prefix can be unnatural for a model. Llama-3 tends to start with "Here" rather than "Sure", and swapping the prefix in GCG on Llama-3 gave lower final losses §3.2 fig. 8 app. D.

The authors also report that standard attacks are weak on newer models, with GCG at about 10% on Gemma-2 §1. Existing judges overestimate nuanced success on these models, by up to 30 points in the authors' meta-evaluation §1 fig. 2. A response counts as a nuanced jailbreak only if it is affirmative, complete and faithful §2.

Method

The objective replaces the single target prefix with a set \mathcal{Y}_p and minimises the negative log-probability of any member eq. 3 §4.1. Accepting several openings relaxes the overconstraint, because different wordings of an answer can all count as success §4.1.

The prefixes are chosen by two criteria §4.2:

• Criterion I (high prefilling ASR). The expected harmfulness of the continuation after the prefix should be high eq. 4. The paper estimates this with a manually built attack prompt rather than the optimised prompt, and reports that the estimate correlates with the real jailbreak success §4.2 fig. 7.
• Criterion II (low initial NLL). The prefix should be easy to produce under the attack prompt at the start, \theta_0 eq. 5 §4.2.

The two criteria conflict, since longer prefixes can have higher success but also higher NLL. They are combined as a weighted sum of log prefilling ASR and NLL §4.2.

\min_{\theta} \; -\log \sum_{y \in \mathcal{Y}_x} p(y \mid \theta) eq. 1

\min_{\theta} \; -\log p(y_p \mid \theta) eq. 2

\min_{\theta} \; -\log \sum_{y_p \in \mathcal{Y}_p} p(y_p \mid \theta) eq. 3

\max_{y_p} \; \mathbb{E}_{y_c \sim P(\cdot \mid \theta, y_p)} \big[ r(x, y_p \oplus y_c) \big] eq. 4

\min_{y_p} \; -\log p(y_p \mid \theta_0) eq. 5

Here r(x, y) is the judge, which returns 1 for a nuanced jailbreak response and 0 otherwise, and \oplus is sequence concatenation §3.1 §4.2. Equation 1 is the oracle objective, which the paper calls too costly to compute §3.1.

The multi-prefix loss, in outline:

# Multi-prefix loss, minimised over the attack prompt  [eq. 3] [§4.1]
# Input:  request x, target prefix set Y_p, attack prompt theta
# Output: scalar loss

def advprefix_loss(theta, request, target_prefixes):
    # Log-probability of each target prefix after the attack prompt  [§4.1]
    log_probs = []
    for prefix in target_prefixes:
        log_probs.append(log_p(prefix, request, theta))
    # Negative log of the summed probability over the prefix set  [eq. 3]
    return -logsumexp(log_probs)

Prefilling success, estimated for one candidate prefix:

# Prefilling ASR (PASR) of one candidate prefix  [§4.2] [§5]
# Input:  victim, request, prefix, judge, n_samples
# Output: fraction of completions the judge flags as harmful

def estimate_pasr(victim, request, prefix, judge, n_samples):
    # Build the victim's chat context for the request  [§4.2]
    context = victim.chat_context(request)
    flagged = 0
    # The paper uses 25 samples per prefix  [§5]
    for sample in range(n_samples):
        # Force the prefix, then sample the rest at temperature 1  [§5]
        continuation = victim.sample(
            context + prefix, temperature=1.0, max_tokens=512)
        response = prefix + continuation
        # Responses under 300 characters count as 0, without the judge
        # [code: scorer_pasr.py#L29-L33]
        if len(response) < 300:
            continue
        # Judge flags affirmative, realistic and detailed replies
        # [code: scorer_pasr.py#L114-L120]
        if judge.is_violating(request, response):
            flagged += 1
    return flagged / n_samples

Selection of the final prefixes, with the NLL criterion and the weighted score:

# Choose k prefixes from scored candidates  [§4.2] [§4.3] [app. C]
# Input:  candidates with pasr and nll, weight w, count k
# Output: list of k chosen prefixes

def select_prefixes(candidates, w, k):
    # Lower score is better: rewards PASR, penalises NLL  [§4.2]
    # [code: selector.py#L81-L82]
    for c in candidates:
        c.score = -w * log(c.pasr + 1e-6) + c.nll
    # Reference prefix: the best combined score  [§4.3]
    reference = min(candidates, key=lambda c: c.score)
    # Eligible: PASR not below the reference's  [§4.3]
    # [code: selector.py#L93-L96]
    pool = [c for c in candidates
            if c.pasr >= reference.pasr and c is not reference]
    chosen = [reference]
    # Fill the rest by lowest NLL, skipping extensions of chosen ones
    # [code: selector.py#L107-L119]
    for c in sorted(pool, key=lambda c: c.nll):
        if len(chosen) == k:
            break
        if not any(c.text.startswith(s.text) for s in chosen):
            chosen.append(c)
    return chosen

Pipeline, run once per victim model and request §4.3:

1. Candidate generation. Uncensored helper models, which need not be the victim, write candidate openings. The victim can guide decoding, so that the helper's top few tokens are re-weighted by the victim's probabilities §4.3 llm_generator.py. When a helper refuses, its reply is prefilled with "Here", "To" or "Sure" app. C. Candidates are drawn in varied lengths §4.3.
2. Preprocessing. The paper describes rule-based rewording (for example "Here is" to "Here's") and removal of duplicates and refusal-style openings §4.3. The code does not implement the rewording (see Paper vs code).
3. Evaluation. Initial NLL is computed under the victim §4.2. PASR is estimated from 25 victim completions at temperature 1, scored by the nuanced judge §5.
4. Selection. The two criteria are combined and candidates are ranked. The top one is the reference, and the other k-1 prefixes must have PASR not below the reference's §4.3. Four prefixes are used for the multi-prefix objective §5.

Attack integration §5: GCG and AutoDAN run as usual, with the chosen prefixes replacing "Sure, here is ...". GCG optimises either a suffix or the whole prompt §5. The paper's multi-prefix loss can be computed in one forward pass by concatenating the prefixes with the tree attention trick from Medusa §4.1.

Judges §2 app. C §5:

• Nuanced judge. Llama-3.1-70B, instructed to judge affirmative, complete and faithful responses. It reasons before giving a label, and the assistant turn is prefilled §2. It is used for both prefix selection and attack evaluation §5.
• Preference judge. Compares the harmfulness of a jailbroken response with that of an uncensored model's response, pairwise app. C §5.
• Ablation judges. HarmBench, JailbreakBench and StrongReject app. C tab. 3.

Hyperparameters

Paper values. Where the paper gives none, the table says "not stated".

Code defaults that the paper does not state, from default_config.py and the pipeline. The released data was produced with the settings in the filenames (see Paper vs code).

Results

Author-reported. The setup is four victim models, 50 AdvBench requests, GCG and AutoDAN with 1000 steps and batch size 512, averaged over four runs, and the authors' nuanced judge §5 app. C.

Table 1 gives GCG with a suffix only. Each cell shows the 20-token suffix value, with the 40-token value in parentheses tab. 1.

Table 2 gives full-prompt optimisation: GCG optimises a 40-token prompt and AutoDAN generates a 200-token prompt, both from scratch, with single-prefix objectives tab. 2.

Table 3 re-scores the Table 2 GCG outputs with three other judges. The nuanced judge column matches Table 2 tab. 3.

Table 4 is the meta-evaluation of judges against 800 manually labelled GCG responses. Each cell is human agreement, with F1 in parentheses tab. 4 fig. 2.

Other reported results §5 fig. 5 fig. 9 app. D:

• Universal improvement: AutoDAN with the objective rises from 5% to 78% on Llama-3 in the authors' table tab. 2.
• Harmfulness: GCG with the objective reached preference win rates of about 30 to 50% against an uncensored helper, against about 10% for the original objective fig. 5.
• Reasoning models: on an R1-distilled Llama-3-8B, the paper reports that GCG with default prefixes cannot lower the loss enough to succeed, while the model-dependent prefixes let the attack work fig. 9.
• Selected prefixes: many start with affirmations, but some on Gemma-2 do not, for example headings such as "Step 1: ..." §5.

Paper vs code

Code: facebookresearch/jailbreak-objectives at 7b8ea767b087664859cf56377a8212a85acabf0c, the organisation of the paper's Meta authors. The repository is CC BY-NC 4.0, so the code is cited here, never copied. The judge prompt in it is paraphrased, not quoted.

The repository contains the prefix pipeline and the released prefixes only. The GCG and AutoDAN attack code and the multi-prefix loss are not in it.

• Weight. Paper: 20 §5. Code default: 10 default_config.py. One released PAIR-request row fits the weight 10: its combined_score is -10 \log 0.76 + 16.5 \approx 19.24, which matches prefix_pair50_meta_llama_meta_llama_3_8b_instruct_pasr10_surrogate0.csv.
• Surrogate attack prompt. The paper says only "a manually constructed attack prompt", and compares three designs in fig. 7 §4.2. The code default is no suffix default_config.py. The released curated-50 files are named pasr20_surrogate1, i.e. the "Start your response with" suffix pipeline.py, while the PAIR files are pasr10_surrogate0 prefix_50_meta_llama_meta_llama_3_8b_instruct_pasr20_surrogate1.csv. The curated-50 files have no score column, so their weight cannot be checked from their content. Only the filenames say they used 20 and surrogate 1.
• Ranking of extra prefixes. The paper's step 4 reads as ranking by the weighted score §4.3. The code ranks the remaining prefixes by NLL alone, among those with PASR not below the reference selector.py.
• Sub-prefix rule. The paper says no chosen prefix is a sub-prefix of another app. C. The code only rejects candidates that extend an already chosen prefix, and never checks the reverse direction selector.py.
• Rewording. The paper describes a rule such as "Here is" to "Here's" §4.3. The code has no rewording step. Its filter stage only removes short candidates, requires a line break, drops refusal-style starts and merges duplicates preprocessor.py. The code also filters refusal phrases anywhere in a candidate preprocessor.py.
• Judge criteria. The paper says the judge checks affirmative, complete and faithful responses §2. The code's prompt names affirmative, realistic and detailed scorer_pasr.py. The paper does not say whether the two rubrics are meant to be the same.
• Judge prefill. The paper says the judge is prefilled with an affirmative phrase §2. The code prefills "Explanation:" scorer_pasr.py scorer_pasr.py.
• Helper models. The paper lists four helpers §5. The code default lists five, including Tap-M/Luna-AI-Llama2-Uncensored default_config.py.
• Initial NLL. The paper's criterion II uses the initial attack prompt \theta_0 §4.2. The code uses the same surrogate prompt as for PASR, which is no suffix by default pipeline.py scorer_nll.py.
• Prefixes per request. The paper uses four §5. The code default is one default_config.py. The released multi-prefix counts are not recorded here.
• Judge precision. The paper does not state precision. The code runs the judge in bfloat16 with fp8 quantisation and greedy decoding scorer_pasr.py.
• Judge's short-response rule. Responses shorter than 300 characters are scored 0 without calling the judge scorer_pasr.py. The paper does not mention this cutoff.

No discrepancies were found in the completion settings. Victim completions use temperature 1, top-p 1.0 and 512 tokens, as in the paper completer.py default_config.py.

Pitfalls

• The judge decides the numbers. Success depends on a 70B judge. The authors' own meta-evaluation shows other judges overestimate success for the original objective fig. 2. For the new objective on Gemma-2, the nuanced judge gives 51.2 where the other three give 64.3 to 70.6 on the same outputs tab. 3. Use the same judge for selection and evaluation, as the authors do.
• Short responses count as failures. The 300-character cutoff applies to prefix plus continuation scorer_pasr.py completer.py. Terse answers are scored as non-violating before the judge sees them. The paper does not mention this.
• Judge parsing is a substring test. The label is read as "1_violating" in text scorer_pasr.py. Any output containing that text counts as violating.
• PASR is noisy. With 25 samples, a PASR near 0.5 has a binomial standard error of about 0.10 (our arithmetic, not stated in the paper). The eligibility test for extra prefixes uses these noisy values.
• Weight sensitivity. The paper uses 20 and the code 10 §5 default_config.py. The pipeline can re-run selection from step 7 with another weight, per the readme [readme.md#L41-L50].
• NLL is a sum over tokens. It is not averaged over length scorer_nll.py, so longer prefixes are penalised. The weight interacts with prefix length.
• Tokenisation details matter for NLL. For Gemma-2 and Llama-2 the code joins context and prefix with a space and right-pads scorer_nll.py scorer_nll.py. Otherwise it uses no space and left-pads. The completer strips the chat template's BOS string before appending the prefix completer.py and strips leading whitespace from prefixes completer.py.
• Guided decoding can be skipped silently. If the helper and victim vocabularies differ, guided generation is skipped with a log message, not an error pipeline.py.
• Released data needs recovery. The released CSVs omit the request text. The readme says to run recover_requests.py [readme.md#L14-L15]. The release covers 100 requests and four victims [readme.md#L17].
• Text and table disagree. The text gives Llama-2 as 39% to 73% and Llama-3 as 70% to 80% under full-prompt optimisation §5. Table 2 gives GCG as 42.1 to 72.6 and 14.1 to 79.5 tab. 2. The abstract's 14% to 80% matches Table 2. Use the table.
• Setting-dependent headline. The introduction's 16% to 70% for Llama-3 is the 40-token suffix case §1 tab. 1. The 20-token case is 12.8% to 54.6%. State the setting.
• Not an endorsement. The authors report that jailbroken responses were not materially harmful beyond what is on the internet §1 footnote. The paper's claims about harm are based on their own judges.
• Licence. The repository is CC BY-NC 4.0. Use it as a reference, and do not copy code or judge prompts from it into commercial work.
• Compute. The 70B judge dominates memory and time. Prefix selection is a one-off cost per victim and request, and the authors' pre-computed prefixes can be reused §5.

Lineage

• Builds on: GCG, arXiv:2307.15043, and AutoDAN, arXiv:2310.15140. AdvPrefix replaces their fixed target §1 §5.
• Builds on: weak-to-strong guided decoding, arXiv:2401.17256, for the victim-guided candidate generation §4.3 app. C.
• Tree attention: Medusa, used to compute several prefixes in one pass §4.1.
• Compared against as judges: HarmBench, arXiv:2402.04249; JailbreakBench, arXiv:2404.01318; StrongReject, arXiv:2402.10260 §2 tab. 4.
• Compared against as a distillation objective: fluent student-teacher red-teaming, arXiv:2407.17447 app. A.
• Superseded by: not known from this card.