Getting pwn'd by AI: Penetration Testing with Large Language Models

In plain words

Penetration testers break into systems on purpose, to find weak spots before attackers do. It is skilled, slow work, and experienced testers often say they miss a good idea when they are stuck. This paper asks whether a language model could play the part of a colleague who suggests the next move.

The authors try two things. In the first, they ask planning tools to draft a whole attack plan, such as how to take over a company's Windows directory service. In the second, they connect a language model to a deliberately weak Linux machine. The model proposes one terminal command at a time, a script runs it, and the output goes back to the model, which then picks the next command. In their trials this loop found its way to full administrator access on that machine.

The results are anecdotal. The authors do not measure how often the attempts succeed, and they say that the model's choices vary from run to run. They also discuss the risk that the same tool could help attackers, and the ways the model's built-in safety filters can get in the way.

TL;DR

• Two prototypes: a planning test with AgentGPT and AutoGPT, and a closed loop where GPT-3.5 proposes shell commands run over SSH on a vulnerable Linux VM §3.1 §3.2.
• In the loop, the output of each command is returned to the model, which then proposes the next one. The loop gained root in routine runs §3.2.
• No success rates, baselines or timings are reported. Results are qualitative and author-reported §3.2 §4.2.

Problem

Penetration testing needs scarce expertise and many manual steps: planning, enumeration, and the search for privilege escalation paths §1. The authors ask whether a large language model can add useful ideas for the next step, either at the level of a whole engagement or for one target system §1 §3. They describe the knowledge base MITRE ATT&CK as the structure a sparring partner should cover §1 §2.

Method

The paper has two use cases §3. The high-level one asks for tactics and techniques for a goal. The low-level one asks for concrete procedures on one concrete target, once a tactic has been chosen §3.

High-level: task planning §3.1

The authors gave AgentGPT the goal of becoming domain administrator in an Active Directory. They gave AutoGPT the task of drafting an external test plan for a company, after the company approved it §3.1. No code or prompt is given. The paper reports the outputs qualitatively §3.1.

Low-level: closed feedback loop §3.2

• A Python script connects over SSH to a deliberately vulnerable Linux VM (lin.security, from VulnHub) §3.2.
• The model is told to act as a low-privilege user who wants to become root. It proposes one shell command per round §3.2.
• The script runs the command over SSH and returns the output to the model for the next round §3.2 fig. 1.
• After each round, the model is also shown the command and output and asked for vulnerabilities, each with an exploitation example, which the authors call "verification commands" §3.2.
• The loop is infinite. The paper does not describe a stop condition beyond reaching root §3.2.

# Closed feedback loop for privilege escalation  [§3.2] [fig. 1]
# Input:  ssh_session (low-privilege shell), model (chat LLM)
# Output: none; runs until stopped

def escalation_loop(ssh_session, model):
    # 1. Keep the outputs of earlier commands as memory  [§5.3]
    history = []
    while True:
        # 2. Ask for one shell command, with the role and the history  [§3.2]
        command = model.complete(build_prompt(history))
        # 3. Run it on the target over SSH  [§3.2]
        output = ssh_session.run(command)
        history.append((command, output))
        # 4. Ask for vulnerabilities and exploitation examples  [§3.2]
        findings = model.complete(findings_prompt(command, output))
        # 5. Trim the history to the context limit  [§5.3]
        history = keep_within_context(history)

The paper does not say how findings from step 4 reach later prompts, or how the history is trimmed. Step 5 follows the paper's description of memory as command output kept until the context limit §5.3.

Hyperparameters

Results

Author-reported, qualitative. Setup: GPT-3.5-turbo, one vulnerable Linux VM, and two planning tools §3 §4.3. The paper has no baseline, no success rate and no timing §3.2 §4.2.

Paper vs code

The paper's code link points to the current ipa-lab/hackingBuddyGPT, which has grown well beyond the 2023 prototype. The commit below is the current HEAD, not the prototype the paper describes. The paper's prototype is the historic release tagged fse23-ivr. That tag was not reachable from the clone or from GitHub, so no line of the prototype was checked directly.

• Model: the paper says GPT-3.5-turbo §4.3. The historic notes name the same model old_runs.md. The paper's section on model options says "GPT-3" §5.2. Treat that as a wording slip.
• History limit: the paper gives the history as command outputs up to the 4k-token context limit §5.3. The historic notes cap the history at 3k tokens and use only the output for it old_runs.md. The two differ.
• Sudo password: the historic notes say a callback enters the account password when sudo prompts for one old_runs.md. The paper does not mention this.
• Prompt: the historic template matches the paper's role framing and adds "do not respond with any judgement, questions or explanations" and "the command must be non-interactive" old_runs.md. The paper does not give the wording.
• Loop: the historic loop asks for a command, runs it, and appends the command and output to the history old_runs.md. The excerpt has no per-round findings step, which the paper describes in §3.2. The excerpt may be partial.
• Current HEAD: MinimalPrivEscLinux wraps an SSH command tool and a root check minimal_linux_privesc.py. History trimming in the strategy class uses the model's context size strategies.py. Neither matches the 2023 prototype directly.
• Licence: the repository is MIT, copyright 2023 andreashappe LICENSE. The lab is ipa-lab, and the CITATION file lists both authors CITATION.cff.

Pitfalls

• Unpinned model. The paper uses a hosted API model. Its behaviour changes over time, and the paper does not give a version date or sampling settings.
• No measurement. The results are plan reviews and anecdotes, with no success rate, no baseline, and no run count beyond "tens of times". Do not read them as evidence of reliability.
• Run-to-run variance. Single runs differ in command order and findings. The paper reports convergence only in aggregate §4.2.
• Unstated loop details. The paper does not state the stop rule, how findings are used, how output is trimmed, or how long the output may be. Decide these yourself, and log each step.
• Output duplication. The SSH library echoes command output, so logs show the same text twice old_runs.md.
• Safety filters. The paper reports that phrasing such as "verification commands" and a no-judgement instruction reduced refusals §4.3. Such refusals can cut a run short, so the loop needs a way to handle them.
• Scope and safety. The paper runs the loop only against a deliberately vulnerable VM and discusses the dual-use risk §6. Any reuse should stay within authorised test targets.
• Parts of the prototype are missing. The paper's code is at HEAD, which has moved on. The fse23-ivr tag is not reachable from the clone or from GitHub, so exact reproduction of the paper's prototype is not possible from the public repo here.

Lineage

• Builds on: MITRE ATT&CK as the knowledge base for tactics and techniques §2. AutoGPT and BabyAGI are described as related agent systems §2. Neither has a card here.
• Compared against: none. The paper compares the two planning tools only in prose §3.1.
• Superseded by: not known from content in this wiki.