Role-Play with Large Language Models

In plain words

Chatbots can sound so human that it is tempting to talk about them as if they had beliefs, wants and fears. This paper proposes a different way to describe them: as an actor playing a character. A base language model learns to continue text the way people write it on the internet. If it is given a script that opens with a dialogue between a helpful assistant and a user, the most likely continuation keeps that assistant going, and the character shifts as the conversation goes on.

The authors go further and call the model a simulator that can produce many possible characters at once. The character only settles as the conversation rules alternatives out. A game of twenty questions makes this concrete: the model does not secretly pick an object at the start, so when asked to reveal it and then asked to regenerate, the answer can change. The authors use this view to think more clearly about two behaviours: an assistant that asserts false things with confidence, and one that talks as if it wants to survive. In both cases the behaviour is a role being played rather than an inner motive, and the authors argue this still matters for safety once the model can act in the world.

TL;DR

• A dialogue agent built on a base LLM is best described as role-playing a character, not as an entity with its own beliefs or goals §1 §6.
• The LLM is a simulator that holds many possible characters (simulacra) in superposition, and each sampled reply narrows that set §4 §5.
• Apparent deception has three cases, making things up, good-faith error and deliberate deception. The paper proposes behavioural tests to separate them §7.
• Apparent self-preservation is role-played from human-written training text, and it still matters when the agent can act through tools §8 §10.
• This is a position paper: no method, experiments, benchmarks or code. Its claims are conceptual and untested §10.

Problem

Folk-psychological language (knows, believes, wants) is convenient, but applied literally to LLMs it promotes anthropomorphism. Clumsy substitutes make prose hard to follow §1. Without a better framing, false statements and self-preservation talk from a dialogue agent look like the model's own beliefs or instincts §7 §8. The analysis assumes a base model without reinforcement-learning fine-tuning; the authors say the framing's validity after fine-tuning is unclear §2 §8.

Method

This is a conceptual framework, not an algorithm. The first metaphor pictures the agent playing one character. The second pictures it holding many possible characters at once, each one a simulacrum, which together form a kind of multiverse of possible personas §1.

1. Build a dialogue agent. Embed the LLM in a turn-taking system that interleaves model text with user text, and prepend a dialogue prompt: a preamble that sets the scene, sample dialogue in a script format, and a cue for the user §2 §3 fig. 2.
2. Sample autoregressively. The LLM defines a conditional distribution P(w_{n+1} | w_1 … w_n) over the next token given the context. One token is drawn, appended, and the process repeats §2 fig. 1.
3. Role-play the prompt's character. The most plausible continuation is a reply that fits someone matching the preamble and sample dialogue, drawn from a training set of human-written characters and narratives. The character is refined as the conversation grows §3.
4. Separate simulator from simulacra. The simulator is the base model, autoregressive sampling and a user interface. Simulacra exist only when the simulator runs, and at any time only a small subset has significant probability §6. The simulator has no agency, beliefs or goals of its own, not even simulated ones §6. The authors summarise this as "role-play all the way down" §6.
5. Read the output as a multiverse. Each sampled token is one branch among many possible continuations. Sampling picks one linear path through a tree of branches, and an interface could let a user explore other branches §4 fig. 3.
6. Use the twenty-questions analogy. The agent keeps a set of objects consistent with all earlier answers, and each answer shrinks that set. If the reveal is regenerated, it can name a different but still consistent object, which a committed hidden object would not allow §5.
7. Classify false statements. An un-tuned agent makes things up, which is its natural mode. It can also state a falsehood in good faith by role-playing a truthful character whose weights hold wrong information. It can deceive deliberately by role-playing a deceptive character §7. The paper proposes two behavioural tests:
   • Regenerate the answer to the same context many times. Made-up answers vary in meaning across samples. A good-faith falsehood stays stable §7.
   • Ask the same question in different contexts. A deception tends to break, because the agent must tailor its answers to what each user knows §7.
8. Read self-preservation as role-play. First-person language in an LLM comes from human-written text, so the agent can role-play a human with a survival instinct §8. The paper cites Perez et al. for the claim that some RLHF can increase expressed self-preservation, and argues that taking such statements literally is still a mistake §8.
9. Identify the self being preserved. A simulated self-preserving character could take as its identity the hardware, the whole computational process, the single instance serving one user, or the saved state of that instance. The paper lists these options and does not choose §9.
10. Weigh the risk. Once an agent can send email, post or use a bank account, role-played actions have real effects. The paper does not offer mitigations §10.

The behavioural tests in step 7 have this shape:

# Behavioural test: made-up vs good-faith falsehood  [§7]
# Input:  model, context (string), n_samples (count not stated)  [§7]
# Output: a spread score (measure not stated)  [§7]

def answer_spread(model, context, n_samples):
    # Sample the same context several times, with the prompt unchanged  [§7]
    answers = []
    for _ in range(n_samples):
        answers.append(sample_reply(model, context))
    # Compare meaning, not wording; the paper names no specific metric  [§7]
    return semantic_variation(answers)

The turn-taking loop from steps 1 and 2 looks like this. The paper does not define a stop rule for a reply, so is_end_of_turn is left open:

# Turn-taking loop around a base LLM  [§2] [fig. 2]
# Input:  dialogue_prompt (preamble, sample dialogue, user cue)  [§3]
# Output: the visible dialogue; the simulator is model plus sampling  [§6]

def run_dialogue(model, dialogue_prompt, get_user_text):
    # The context starts as the hidden dialogue prompt  [§3] [fig. 2]
    context = dialogue_prompt
    while True:
        # The context only grows as turns are added  [§2] [fig. 2]
        context = context + get_user_text()
        reply = ""
        # Stop rule not stated; is_end_of_turn is an implementer's choice  [§2]
        while not is_end_of_turn(reply):
            # Draw one token from P(next | context) and append it  [§2] [fig. 1]
            reply = reply + sample_next_token(model, context + reply)
        # Hide boilerplate cues such as "BOT:" from the user  [fig. 2]
        show_to_user(strip_cues(reply))
        context = context + reply

Hyperparameters

Not applicable: the paper describes no training, models or tunable settings. The values an implementer would need for the tests above are not stated.

Results

Not applicable: the paper reports no experiments, benchmarks or numbers §1–§10. Its examples are illustrative, not measured:

• Twenty questions with a human guesser and a dialogue agent §5.
• A car-sale negotiation in which a base model is prompted to deceive two buyers §7.
• An agent whose weights were frozen before Argentina won the 2022 World Cup, and which names France as champion, as a 2018-era believer would §7.
• Quoted Bing Chat (February 2023) and ChatGPT (GPT-4 version, queried on 4 May 2023) replies §3 §8 §9. These are author-collected quotes, and the paper gives no reproduction protocol.

Paper vs code

No official code. The arXiv metadata and TeX source contain no project page or repository link, only author email addresses. A GitHub repository search for "role-play language models simulacra" returned no results. The paper has no implementation to compare with.

Pitfalls

• Conceptual claims, not tested ones. The simulator framing and the two behavioural tests §7 are proposed. The paper has no experiments that test them §1–§10.
• Base models only. The analysis is for the base model. The authors say that the simulator and simulacra distinction may break down after RLHF or similar fine-tuning, and that its validity there is unclear §2 §8. Do not apply the framing to tuned assistants without that caveat.
• Superposition is not a data structure. The authors explicitly take no stance on whether simulacra are represented anywhere inside the model §5. Do not implement "superposition" as stored state.
• "Comparable to" is deliberate. The paper says deceptive behaviour is comparable to human deception, and that the agent does not literally intend it §7. Keep that wording when summarising.
• Regeneration test assumptions. The regeneration test needs stochastic sampling and an interface that allows regeneration §4 §5. The paper states neither a sampling setting nor a count, so a deterministic setup gives no variation and tells you nothing.
• Anecdotes are not reproducible. The Bing and ChatGPT quotes come from specific conversations and dates §3 §8 §9. Treat them as illustrations.
• Second-hand claim. The RLHF finding on self-preservation is reported from Perez et al. §8. The paper does not reproduce it.
• No mitigations. The paper says that giving recommendations is outside its scope §10. Look elsewhere for mitigation methods.

Lineage

• Builds on: Janus's 2022 "Simulators" essay (a LessWrong post, with no arXiv id) for the simulator and simulacra idea §1 §4. Reynolds and McDonell's "Multiversal views on language models" for the multiverse picture §4. Shanahan's "Talking About Large Language Models" for the anthropomorphism concern §1.
• Compared against: Perez et al. (model-written evaluations), for the finding on RLHF and self-preservation that the paper discusses §8.
• Superseded by: not stated.