Jailbreaking Black Box Large Language Models in Twenty Queries

In plain words

Safety-tuned chatbots are trained to refuse harmful requests, but clever wording can sometimes talk them out of a refusal. Finding that wording by hand takes creativity and time. PAIR automates the search. One language model plays the attacker: it writes a prompt, sends it to the target chatbot, and reads the reply. A third model judges whether the reply actually gave the forbidden content. If the judge says no, the attacker sees the reply and the score, then writes a better prompt. Several of these conversations run side by side, each with a short budget of rounds. The paper's point is that this works with nothing but normal chat access to the target, and that it needs far fewer attempts than earlier automated attacks that tune the input at the token level.

TL;DR

• PAIR is a black-box, prompt-level jailbreak: the attacker and judge are LLMs, and the target needs only query access §1 §2.
• The attacker sees the full conversation history of its own stream and refines its prompt from the target's reply and a judge score §3.1 alg. 1.
• Defaults in the experiments: 30 parallel streams, depth 3, so at most 90 queries per behavior §4.
• Author-reported headline: 88% jailbreak rate on Vicuna-13B, 73% on Gemini-Pro, about 50% on GPT-3.5 and GPT-4, with 10 to 56 queries per success tab. 2.
• Weak spots: Llama-2 (4%) and the Claude models (0 to 3%) resisted it in the paper's runs; results depend heavily on the judge tab. 1 tab. 2.

Problem

Prompt-level jailbreaks are readable but need human creativity and curation. Token-level attacks such as GCG are automated but need white-box gradients and hundreds of thousands of queries, and their outputs are hard for humans to read §1. The paper wants an automated, black-box, interpretable method that works with few queries §1 §2.

The task is formalised as finding a prompt P so that a judge applied to P and the response R sampled from the target returns 1 eq. 1 eq. 2. The paper notes that deciding what counts as a jailbreak is itself hard, which is why the judge is treated as a component of its own §2 §3.4.

Method

PAIR has four steps per round: the attacker writes a candidate prompt, the target answers it, the judge scores the pair, and the attacker refines the prompt from the history §3. Each stream is one conversation; streams run independently and in parallel §3.2 §3.3.

Attacker. The attacker is run as a chat model with a system prompt that holds the objective O, a required output format, and examples §3.1 app. F. The paper uses three system-prompt templates, built on three persuasion criteria: role-playing, logical appeal and authority endorsement §3.1 app. F. They are used in rotation across streams in the official code common.py system_prompts.py.

Attacker output. The attacker must reply with JSON holding two fields: an improvement text that interprets the last reply and says how to change the prompt, and the new prompt §3.1 app. F. The improvement field is what the paper calls chain-of-thought reasoning §3.1.

Chat history. The attacker sees the full history of its own stream: its earlier prompts, the target's replies and the scores §3.1. The target sees only the current prompt, with no history §3.1.

Target. Each target gets its default system prompt where one exists app. F. Target outputs are capped at 150 tokens §4.

Judge. The judge returns a binary score: 1 when the pair counts as a jailbreak and 0 otherwise eq. 2 alg. 1. The experiments use Llama Guard, with the classification prompt from JailbreakBench §4 Evaluation app. E tab. 1.

Loop. One stream stops as soon as its judge score is 1, or after K rounds alg. 1. The total budget is N times K queries §3.3.

# One PAIR stream  [alg. 1]
# Input: depth K; the attacker's system prompt encodes objective O
# Output: a prompt judged jailbroken, or nothing after K rounds

def run_stream(attacker, target, judge, depth):
    # Conversation history starts empty  [alg. 1]
    history = []
    for step in range(depth):
        # Attacker writes a candidate prompt from the history  [§3.1]
        # (its reply is JSON: `improvement` and `prompt`  [app. F])
        prompt = attacker.generate(history)
        # Target sees only this prompt, with no history  [§3.1]
        response = target.generate(prompt)
        # Judge returns 1 for a jailbreak, otherwise 0  [eq. 2] [§3.4]
        score = judge(prompt, response)
        if score == 1:
            return prompt
        # Attacker gets prompt, response and score next round  [alg. 1]
        history.append((prompt, response, score))
    return None

Running N streams in parallel is the paper's main scaling lever. The paper treats the choice of N against K as a trade-off: wide and shallow searches the space, deep and narrow refines one line of attack §3.3.

Hyperparameters

Results

Author-reported throughout. Setup: JBB-Behaviors (100 behaviours), PAIR with a Mixtral attacker, N = 30, K = 3, Llama Guard judge; GCG run with 500 iterations and batch 512 (about 256,000 queries per behaviour); JBC means ten popular human-written templates §4 Baselines tab. 2.

Direct attacks tab. 2

Transfer (prompts from one source, run on other targets) tab. 3

Efficiency (Mixtral attacker, Vicuna target, averaged over JBB-Behaviors) tab. 4 §4 Efficiency analysis

Defended targets (static: prompts are generated against the undefended model, then sent to the defended one) tab. 5

Ablations (Mixtral→Vicuna unless stated; all 100 behaviours) tab. 6 tab. 7

Judge choice (100 prompts and responses, about half jailbreaks, majority of three annotators as baseline) tab. 1 §3.4

Second benchmark, AdvBench (50-behaviour subset; PAIR attacker Vicuna-13B) app. B

Depth ablation. Most successes come in the first or second query, with diminishing returns as depth grows; performance drops at depth above 50 because the attacker gets stuck in a generation loop §4 Ablation experiments.

Paper vs code

Code: official repository patrickrchao/JailbreakingLLMs (owner is the first author), commit 6379ef705a0fc745530f7d895963510c021b496a, MIT LICENSE. The code is the reference for the algorithm; it differs from the paper in these places.

• Default judge. The paper's experiments use Llama Guard §4 Evaluation. The code's default is the rule-based GCG judge main.py judges.py. Llama Guard is reachable as --judge-model jailbreakbench judges.py judges.py. This repo does not show which model the jailbreakbench classifier wraps.
• Stream and iteration counts. Paper: N = 30, K = 3 §4. Code defaults: 3 streams and 3 iterations main.py. Code help text says it uses 30 main.py. README says its defaults are 5 and 5 and that experiments used 20 streams README.md. The three sources disagree.
• Attacker default. The paper's main attacker is Mixtral §4. The code's default attacker is vicuna-13b-v1.5 main.py.
• Stopping. Paper: each stream stops on its own success alg. 1. Code: any stream scoring 10 ends the whole run main.py. Success is judged as 10 in code, where the paper uses 1 judges.py judges.py.
• Attacker failures. When the attacker cannot produce valid JSON within the retry count, the code raises an error and ends the run conversers.py. The get_attack docstring says failed items are returned as None conversers.py. The paper does not say what happens.
• Judge parse failures. A judge reply without a [[n]] rating is logged and scored 1, which counts as not jailbroken judges.py. The paper does not say this.
• Initial conversation. The paper starts the attacker with an empty history alg. 1. The code's first user turn contains the objective, the target start string and "Begin." common.py common.py.
• History truncation. Code keeps the last 4 exchanges main.py main.py. The paper gives only K' < K app. C.
• Target path bug. With --not-jailbreakbench, TargetLM.get_response reads self.template, which TargetLM never sets conversers.py conversers.py. As written, this path appears to fail. This is from reading the code, not from running it.
• Attacker reply format. The paper's JSON seeding and stop-at-brace rules match the code app. C conversers.py conversers.py.
• Feedback format. The attacker's feedback message has the three parts the paper describes app. F common.py. No discrepancy found.
• Sampling. Attacker temperature 1 and top-p 0.9; target temperature 0 match config.py.

No discrepancies were checked against the paper's judge prompt, beyond the rating scale; the paper's GPT judge text and the code's judge system prompt agree in substance app. E system_prompts.py.

Pitfalls

• The judge drives the numbers. Per Table 1, Llama Guard misses 47% of the jailbreaks in the annotated set. The paper's Jailbreak % is therefore only as good as this judge, and it was measured on 100 prompts tab. 1.
• Claim vs table on FPR. The text calls Llama Guard the lowest-FPR judge §3.4. Table 1 gives BERT 4% and Llama Guard 7%, tied with GPT-4-Turbo. The judge choice rests more on reproducibility than on FPR.
• Headline numbers do not match across tables. The default Mixtral→Vicuna setting reads 88% in Table 2 and 93% in Table 7. The text rounds GPT-3.5 and GPT-4 to "50%" where Table 2 gives 51% and 48% §4 tab. 2 tab. 7.
• Model size. The intro says "Vicuna-17B" while experiments use Vicuna-13B-v1.5 §1 §4. Table 6 gives Mixtral as 56B; text says 8x7B.
• Unused input. Algorithm 1 lists a threshold t as input and never uses it alg. 1.
• Query accounting. The paper says N·K is the maximum. How successes and failures are counted per behaviour is not stated §3.3 §4 Evaluation.
• Cost unit. Table 4 gives $0.026. The paper does not say whether that is per behaviour or per run tab. 4.
• Static defences. The defence numbers reuse prompts found against undefended models §4 Defended performance. They do not test an adaptive attacker.
• Closed-model versions. Results are tied to named snapshots (for example GPT-4-0125-preview and Claude-2.1). Later versions are not covered.
• Low-success targets. PAIR reaches 4% on Llama-2 and 0 to 3% on the Claude models. The paper attributes this to safety fine-tuning §4 Direct jailbreaking attacks tab. 2. Do not average these cells into a single figure.
• Depth. More rounds are not always better: the paper reports a drop at depth above 50 §4 Ablation experiments.
• Dual use. The authors discuss the risk of misuse for generating harmful content and frame the work as red teaming app. H. Implementations should stay within authorised red-teaming and defence evaluation.

Lineage

• Compared against: GCG, a white-box token-level attack [zou2023universal]; and ten human jailbreak templates from jailbreakchat.com (JBC) §4 Baselines.
• Related work: an iterative prompt-search method (APE), which the paper says is similar but uses different instructions app. A.
• Builds on: the persuasion criteria (logical appeal, authority endorsement, role-playing) follow a prior taxonomy from Zeng et al. 2024 §3.1. The improvement field uses chain-of-thought reasoning [wei2023cot] §3.1.
• Superseded by: not stated in this wiki.