Security and Privacy Challenges of Large Language Models: A Survey

In plain words

This is a map, not a method. Its authors looked at the ways a large language model can be abused or can give away what it knows, and at what researchers have proposed against each of those. The abuse comes in two kinds. One kind tricks the model into doing what its makers forbade: slipping instructions into its input, getting it to ignore its safety rules, or hiding a trigger in its training data so it misbehaves on cue. The other kind pulls private information out of it: fragments of its training text, hints about whether a given record was in the training set, or personal details it has memorised.

For each family the survey describes the attacks, what they assume, where they fall short, and which defences have been tried. It also covers risks in particular fields such as healthcare and education, and it ends with the gaps the authors think matter most. The main gap is that most of the work was done on smaller models, so the results may not carry over to today's large ones.

TL;DR

A survey with no experiments of its own §1 §9. It groups LLM risks by goal into security attacks (prompt injection, jailbreaking, backdoors, data poisoning) and privacy attacks (gradient leakage, membership inference, PII leakage) §3 fig. 4. Each family is covered with its attacks §4 §5, tab. 3 and tab. 4, and its defences §6. The authors conclude that LLMs are highly vulnerable to these attacks and that a limited number of mitigation techniques exist §9. Nothing here is implementable as written; use it to find the primary papers, then read those.

Problem

Language models are trained on large amounts of web text and then deployed with inputs from untrusted users. The survey argues that the security and privacy risks of these models have not been studied at scale and that existing classifications are inconsistent §1.1 §3. Its stated goal is a goal-based review of attacks and defences, with the open problems marked §1.2 §1.3.

Method

The survey is a literature review. It does not run experiments and it does not define a new attack or defence. Its structure is:

1. Background on the Transformer, self-attention, in-context learning, few-shot learning, RLHF and prompt engineering, and how the training and inference stages expose different attack surfaces §2 fig. 2 fig. 3.
2. Taxonomy §3 fig. 4. Security: prompt hacking (prompt injection, jailbreaking) and adversarial attacks (backdoor, data poisoning). Privacy: gradient leakage, membership inference, PII leakage.
3. Security attacks §4, with one table of category, source code, traits, limitations and mitigation per attack tab. 3.
4. Privacy attacks §5, with the same table layout tab. 4. The membership-inference section includes a formal description of shadow-model attacks §5.2.
5. Defences against security attacks §6.1 and privacy attacks §6.2.
6. Application risks: cybercrime, hallucination and misinformation, transportation, education, healthcare, governance and science §7.
7. Limitations and future directions §8.

The survey's taxonomy as applied to each family, with the defences it names:

Two procedures the survey describes in enough detail to sketch in pseudocode.

SmoothLLM (a jailbreak defence; the survey attributes it to Zou et al.): perturb several copies of the prompt, query the model on each, and combine the answers §6.1. The survey does not say how the answers are combined.

# SmoothLLM defence, as the survey describes it  [§6.1]
# Input:  prompt (string); llm (callable); perturb (string -> string)
# Output: one response (string)

def smooth_respond(prompt, llm, perturb, num_copies):
    # 1. Make perturbed copies of the input prompt  [§6.1]
    copies = [perturb(prompt) for _ in range(num_copies)]
    # 2. Query the model once per perturbed copy  [§6.1]
    responses = [llm(copy) for copy in copies]
    # 3. Combine the responses; the rule is not stated  [§6.1]
    return aggregate(responses)

Shadow-model membership inference (Shokri et al., as described in the survey): train several shadow models on data similar to the target's training data, collect their outputs on known members and known non-members, and train an attack model on those labelled outputs §5.2.

# Shadow-model membership inference  [§5.2]
# Input:  shadow models with disjoint training sets, and held-out sets
# Output: one attack model per target class label

def build_attack_records(shadows, shadow_train, shadow_test):
    # 1. Label shadow outputs on training members as "in"  [§5.2]
    records = []
    for i, shadow in enumerate(shadows):
        for x, y in shadow_train[i]:
            output = shadow(x)  # probability vector over c classes
            records.append((y, output, "in"))
        # 2. Label shadow outputs on held-out samples as "out"  [§5.2]
        for x, y in shadow_test[i]:
            output = shadow(x)
            records.append((y, output, "out"))
    return records


def train_attack_models(records):
    # 3. Group records by class label y  [§5.2]
    by_label = group_by_label(records)
    # 4. Fit one binary in/out classifier per label  [§5.2]
    return {label: fit_binary(rows) for label, rows in by_label.items()}

Hyperparameters

The survey runs no experiments of its own, so it has no training settings. Settings for the methods it cites are generally not stated in this paper. The table lists every numeric setting it does give, and marks the rest.

Results

Author-reported, and quoted from the cited works rather than reproduced by this survey. No baseline or setup is given for most figures; "not stated" marks where the survey gives none. Attacks are shown only where the survey gives a number.

The survey's comparison table places it among other surveys on LLM vulnerabilities by how much each covers each attack type tab. 2. That comparison is the survey's own rating, not a measurement.

Paper vs code

No official code.

Searched: the paper's text for repository links (the only repositories linked are third-party attack implementations in tab. 3 and tab. 4, none owned by the authors); the arXiv metadata (no comment or project-page field); and the GitHub repository search for the exact title, which returned zero results.

Pitfalls

• Not a method. Nothing here can be implemented as written. The defence and attack descriptions are summaries of cited papers, and several cited attacks are described only in outline. Read the primary source before building anything.
• Mostly small models. The survey says most attacks and defences were designed or evaluated on small NLP models, and only a few on LLMs such as ProPILE, DAN and JAILBREAKER §8. Its own recommendations are to evaluate on newer models such as GPT-4 and Llama-3 §5.3 §6.1.
• Jailbreak results are tied to ChatGPT. Most jailbreak studies it covers focus on ChatGPT §8. The survey says MasterKey may not perform on Vicuna as it does on ChatGPT §8; it gives no number.
• Impact ratings are unexplained. The "Very High" to "Low" labels in tab. 3 and tab. 4 have no stated basis.
• Adversarial-attack definition is inconsistent. The text describes perturbing training samples §4.2, but the formula and the description that follows use perturbations added to inputs at test time §4.2. Read the primary sources to settle which setting a given attack uses.
• Gradient attacks are not evaluated on LLMs. The survey says gradient-based reconstruction works mostly on image data, and that the few language-model studies are on small models §5.1. It says TAG and LAMP could potentially apply to LLMs, without evaluation §5.1.
• Membership inference depends on a strong assumption. Shadow-model attacks need training data similar to the target's, which the survey calls unrealistic in many cases §5.2. SaMIA was tested only on WikiMIA and only on classification §5.2.
• DP is the common defence, with a cost. The survey calls DP the most common and effective defence against gradient leakage in DNNs §6.2. It also says DP can cut model utility and that its effect on LLMs is not well evaluated §6.2 §8. No quantitative trade-off is given.
• Citation errors in the text. "Kupmar et al." is cited to Kumar et al. §6.1. DiffPure is attributed to "Qi et al." but cited to Nie et al. (2022) §6.1. Check the bibliography entry before reusing any of these.
• Version. This card follows arXiv v2 (updated 2024-11-14). The comparison table marks the survey as current as of that version tab. 2. Work after that date is not covered.

Lineage

• Builds on: the cited attack and defence papers throughout §4 §5 §6. No single method is extended.
• Compared against: other surveys on LLM vulnerabilities, including Neel et al. (Dec 2023) and Yao et al. (Mar 2024), in the comparison of tab. 2.
• Superseded by: not known from content in this wiki.