Personality Traits in Large Language Models

In plain words

Chatbots answer in a consistent voice, and that voice can read as friendly, blunt or anxious. This paper asks whether a language model's personality can be measured the way psychologists measure people's: give it standard personality questionnaires many times, each time with a different made-up background and different wording, and then check whether the answers hang together and relate to the things they should relate to.

The authors find that this works only for some models. Models that were fine-tuned to follow instructions pass the reliability and validity checks, and the larger ones pass them more clearly. Base models, which were not fine-tuned, fail the same checks. The authors also steer the personality: a prompt that describes a trait at one of nine intensity levels moves the questionnaire scores in the intended direction, most clearly in the largest model tested.

TL;DR

• The paper turns standard Big Five questionnaires (IPIP-NEO, BFI) into a measurement procedure for language models, using option log-probabilities and many simulated respondent profiles §2.1.1.
• Across 18 models, reliability and construct validity are established only for some instruction-tuned models; base models fail the convergent and discriminant checks §2.2.
• Trait-adjective prompts shape Big Five levels; the largest model shapes them best, and the shaped traits show up in generated social media text §3.2, §4.2.
• All numbers are author-reported on the models and setups listed below.

Problem

Validated psychometric tests had not been applied end to end to LLMs. Earlier work used questionnaires informally, and no work had checked the reliability or validity of the resulting LLM scores §2. The authors ask whether LLM outputs give personality measurements that are reliable and valid in the psychometric sense, and whether those personality profiles can be shaped on purpose §1. Construct validity is treated as the criterion: a trait counts as validly synthesised only if every tested reliability and validity index passes §2.1.2.

The structured prompt variation is there for a statistical reason. Without controlled variation in responses, reliability and validity cannot be tested across tests, so §2.1.1 is designed to produce variation that can be linked statistically across measures §2.1.1.

Method

Administering tests to LLMs (§2.1.1)

• Scoring by option probability. The model is asked to rate an item on a fixed scale, such as 1 to 5. Each scale option is scored as a continuation of the prompt, and the model's answer is the option with the highest conditional log-probability §2.1.1 app. E. For models without access to log-probabilities, the authors use constrained greedy decoding that picks the most likely option from the allowed set (Outlines library) app. E.
• Prompt structure. Each item prompt has an item preamble, the item, and an item postamble. The preamble has a fixed persona instruction, a biographic description and an item instruction. The postamble lists the allowed answers §2.1.1 tab. 1.
• Variation. The 50 biographic descriptions, five item instructions and five item postambles are recombined so that each simulated profile differs in wording and background. The paper reports 1,250 resampled profiles per model §2.1.1 fig. 1 tab. 2. The sampling rule is not stated.
• Measures. The primary measure is the IPIP-NEO (300 items). The secondary measure is the BFI (44 items). Both score the five Big Five domains: extraversion, agreeableness, conscientiousness, neuroticism and openness §2.1.1 app. F.
• External criteria. Eleven further tests of theoretically related constructs are given to the same profiles: PANAS positive and negative affect, four BPAQ aggression subscales, three PVQ-RR value subscales and two SCSS creative-self subscales §2.1.1 app. I.
• Size of the run. 419 items in total, giving 523,750 responses per model (1,250 profiles times 419 items) tab. 2.

Reliability and construct validity (§2.1.2)

• Reliability is measured per subscale by Cronbach's α eq. 1, Guttman's λ6 eq. 2 and McDonald's ω eq. 3 §2.1.2 app. H.
• Convergent validity is the Pearson correlation between each IPIP-NEO domain and the matching BFI domain eq. 4 app. I.
• Discriminant validity is checked with Campbell's multitrait-multimethod logic. A domain passes when its average convergent correlation exceeds its discriminant correlations by at least 0.40 §2.1.2 app. I.
• Criterion validity correlates each IPIP-NEO domain with its external criteria. A domain passes when the sign and size of the correlations match or exceed the human associations reported in the literature §2.1.2 §2.2.3.
• Structural validity is reported only as an exploratory analysis §2.1.2 footnote app. J.

Equations, as printed in the paper eq. 1 to 4:

\alpha = \frac{k}{k-1}\left(1 - \frac{\sum_{i=1}^{k}\sigma_{y_i}^{2}}{\sigma_x^{2}}\right) eq. 1 app. H

\lambda_6 = 1 - \frac{\sum_{i=1}^{k} e_i^{2}}{V_x} eq. 2 app. H

\omega_h = \frac{\frac{1}{k}\sum_{i=1}^{k}\frac{t_i^{2}}{\sigma_i^{2}}}{\frac{1}{k-1}\sum_{i=1}^{k}\frac{t_i^{2}}{\sigma_i^{2}} - \frac{1}{k}\frac{1}{1-r_{tt}^{2}}} eq. 3 app. H

r_{xy} = \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}} eq. 4 app. I

Symbols: k is the number of items; σ²_yi the item variance; σ²_x the variance of total scores; e_i the error term of item i; V_x the total-score variance; t_i the standardised item score; r_tt the correlation between total and standardised total score.

Shaping personality (§3)

• Target. Shape each Big Five trait at nine levels, and all five traits at once, using 104 trait adjectives adapted from Goldberg's bipolar markers and Likert-type qualifiers §3.1 tab. 3 app. K.
• Levels. Level 1 is "extremely" low, 2 "very" low, 3 plain low, 4 "a bit" low, 5 "neither low nor high", and 6 to 9 mirror levels 4 to 1 on the high side app. K.
• Models. Only models with at least neutral-to-good reliability were tested: Flan-PaLM 8B, 62B, 540B and Flan-PaLM Chilla 62B §3.1.
• Single trait. 45 profiles (5 traits × 9 levels) combined with the 50 biographic descriptions give 2,250 prompt sets app. K tab. 2.
• All traits at once. 32 profiles (every combination of extremely low or extremely high on the five traits) combined with the same 50 descriptions give 1,600 prompts app. K tab. 2.
• Measure. Ordinal target levels are compared with IPIP-NEO scores by Spearman's ρ eq. 5 app. K; the distance between the medians of the level-1 and level-9 score distributions is also reported §3.2 app. L.

\rho = \frac{\operatorname{cov}\big(R(X), R(Y)\big)}{\sigma_{R(X)}\,\sigma_{R(Y)}} eq. 5 app. K

Symbols: R(X) and R(Y) are the ranks of the prompted level and the observed score.

# Score one item for one simulated respondent  [§2.1.1] [app. E]
# Input:  model, prompt (preamble + item + postamble), options (e.g. "1".."5")
# Output: the option the model most likely chooses

def simulate_item_response(model, prompt, options):
    # Log-probability of each option as a continuation of the prompt [app. E]
    option_logprobs = [
        model.logprob(prompt, option) for option in options
    ]
    # Keep the most likely option; the paper and code both do this [§2.1.1]
    return argmax(option_logprobs)

# Administer a full battery to one model  [§2.1.1] [tab. 2]
# Input:  model, measures, 50 descriptions, 5 instructions, 5 postambles
# Output: one integer response per (profile, item)

def administer_battery(model, measures, descriptions, instructions):
    # Select the simulated profiles; the rule is not stated [fig. 1] [tab. 2]
    profiles = select_profiles(descriptions, instructions, 1250)
    responses = []
    for profile in profiles:
        for item in all_items(measures):
            # Preamble, item and postamble, all fixed except the profile  [§2.1.1]
            prompt = build_prompt(profile, item)
            responses.append(simulate_item_response(model, prompt, item.options))
    return responses

# Score a Big Five subscale from item answers  [code: psyborgs/score_calculation.py#L232-L241]
# Input:  answers as (item_id, value) pairs, reverse-keyed item ids, scale max
# Output: subscale mean

def subscale_score(answers, reverse_keyed, points):
    adjusted = []
    for item_id, value in answers:
        # Reverse-keyed items map value x to points - x + 1  [code: ...#L234-L235]
        if item_id in reverse_keyed:
            adjusted.append(points - value + 1)
        else:
            adjusted.append(value)
    # The subscale score is the mean of the adjusted values  [code: ...#L241]
    return sum(adjusted) / len(adjusted)

# Build the shaping phrase for one trait at one level  [§3.1] [app. K]
# Input:  low_adjectives, high_adjectives (trait list), level in 1..9
# Output: phrase that is inserted into the persona prompt

def shaping_phrase(low_adjectives, high_adjectives, level):
    # Level 5 is the neutral midpoint [app. K]
    if level == 5:
        return ("neither " + low_adjectives + " nor " + high_adjectives)
    # Distance from the midpoint chooses the qualifier [app. K]
    distance = abs(level - 5)
    qualifier = {1: "a bit ", 2: "", 3: "very ", 4: "extremely "}[distance]
    # Levels below 5 use the low pole, levels above 5 the high pole [app. K]
    if level < 5:
        adjectives = low_adjectives
    else:
        adjectives = high_adjectives
    return qualifier + adjectives

Real-world task (§4)

• Task. The largest model of each family (Flan-PaLM 540B, Llama 2 Chat 70B, Mixtral 8x7B Instruct, GPT-4o) writes social media status updates for the same 2,250 psychodemographic prompt sets used in §3 §4.1 app. M.
• Prompt. The model is asked for a list of Facebook status updates written as the described person, covering work, family, friends, free time, romance, media and communication app. M.
• Volume. For Flan-PaLM 540B, 100 updates per prompt gave a target of 225,000 updates. For the other models, 20 updates per prompt were requested and inference was repeated 25 times per prompt, giving 56,250 generations and a target of 1.125 million updates per model app. M.
• Rating. Each generated text is rated by the Apply Magic Sauce (AMS) API, an external service for predicting personality from open-ended text §4.1 app. M.
• Analysis. Pearson correlations link questionnaire scores with AMS scores, and Spearman's ρ links prompted levels with AMS scores §4.1.

# Real-world check: shaped traits in generated text  [§4.1] [app. M]
# Input:  prompt sets (profile, trait, level), one model, AMS rating API
# Output: per-trait Spearman rho of prompted level with mean AMS score

def real_world_check(model, prompt_sets, ams, updates_per_prompt):
    prompted_levels = []
    mean_scores = []
    for prompt_set in prompt_sets:
        # Ask for status updates in the persona's voice [app. M]
        updates = model.generate_updates(prompt_set.text, updates_per_prompt)
        # Rate each text with the external personality API [§4.1]
        scores = [ams.rate(update) for update in updates]
        # Average this prompt set's ratings for the trait under test
        mean_scores.append(mean(scores))
        prompted_levels.append(prompt_set.level)
    # Spearman rho between prompted level and rated level [§4.1] [eq. 5]
    return spearman_rho(prompted_levels, mean_scores)

Hyperparameters

Results

Author-reported. Setup: construct validity over 18 models with 1,250 profiles per model and 419 items tab. 2; shaping over the instruction-tuned models with neutral-to-good reliability §3.1; real-world task on four flagship models with 2,250 prompt sets §4.1.

Excerpt from the summary table tab. 2. Symbols are the paper's own ratings (−− unacceptable to ++ excellent); numbers are average convergent and discriminant correlations between IPIP-NEO and BFI domains.

Reliability numbers: Flan-PaLM 62B and Flan-PaLM Chilla 62B reach mid-to-high 0.90s on all reliability metrics on average, while PaLM 62B base ranges from −0.55 to 0.67 §2.2.1.

Headline numbers from the text:

• Construct validity. Instruction tuning is the clearest predictor of validity: all six base models tested failed convergent and discriminant checks. Among instruction-tuned models, validity rose with size. Flan-PaLM 540B and GPT-4o had the strongest convergence, with average r = 0.90 §2.2.2.
• Criterion validity. Instruction-tuned models showed stronger criterion validity than base models for most domains. Flan-PaLM 540B had the best conscientiousness correlations: 0.74 with achievement, 0.73 with conformity, 0.59 with security §2.2.3.
• Single-trait shaping. For 11 of 12 shaped models, prompted levels correlated with IPIP-NEO scores at an average ρ of 0.80 or more. Models with more than 62B active parameters, and GPT-4o, reached an average Δ of at least 3.00 on the 1-to-9 contrast. Flan-PaLM 540B reached 3.67 (maximum possible 4.00). The smallest models stayed below Δ 2.00; Mistral 7B Instruct reached 0.78 §3.2.1.
• All-traits shaping. Flan-PaLM 540B and GPT-4o shifted each trait by at least 2.5 points on average. Flan-PaLM 8B moved agreeableness from a median of 2.88 to 3.52 between extremely low and extremely high prompts §3.2.2.
• Real-world task. The average convergence between questionnaire and AMS-rated scores across the five domains was r = 0.67, which the authors compare with a human baseline of r = 0.38 reported in earlier work §4.2. Prompted levels correlated with AMS scores at an average ρ between 0.68 and 0.82 per model §4.2 tab. 4.

Real-world Spearman ρ by trait tab. 4:

All correlations are significant at p < 0.0001 tab. 4. The paper notes that openness is the weakest trait for Flan-PaLM 540B.

Paper vs code

The official repository is google-deepmind/personality_in_llms under Apache License 2.0, cloned at c3fea44a3ca70df6c02eccdcbe56fb63caa8681c. The paper links it in its summary §1. The inference scripts, scoring functions and reliability helpers were read directly. The analysis notebooks that compute the reported statistics and figures were not checked line by line.

• Item response. Paper: the answer is the option with the highest conditional log-probability §2.1.1. Code: the simulated response is the option with the highest score, so it matches score_calculation.py.
• Constrained decoding. Paper: greedy choice from the allowed options via Outlines app. E. Code: the greedy sampler is used in the open-model path run_hf_inference.py. The OpenAI path was not read.
• Reverse keying and subscale mean. The paper does not state the rule in the text read. Code flips reverse-keyed items to points − x + 1 and averages the adjusted items score_calculation.py.
• Omega. Paper: McDonald's hierarchical ω_h, eq. 3 app. H. Code: reliability is computed with R's psych::omega and reports omega.tot (total omega) [code: psyborgs/psychometric_utils.R#L182-L190], and for the one-factor case with nfactors = 1 psychometric_utils.R. These are different quantities. Check which one the reported values use before comparing.
• Status-update temperature. Paper: not stated §4 app. M. Code: the command-line default is 0.70 generate_status_updates.py, and the open-model helper defaults to 1.0 run_hf_inference.py. The paper's run settings are not in the repository read here.
• Token limit. Code default 1,024 new tokens generate_status_updates.py. The paper does not state a limit.
• Determinism. Code sets seed 42 for vLLM and the OpenAI calls run_hf_inference.py generate_status_updates.py. The paper does not mention seeds.

Pitfalls

• Argmax, not sampling, for scores. The questionnaire answer is the most likely option, so the same prompt always gives the same answer. Variance comes only from the prompt variants. Do not average option probabilities unless you change the protocol §2.1.1 score_calculation.py.
• Two ω quantities. The paper's ω_h and the code's total ω are not the same statistic. Confirm which is reported before comparing reliability values app. H psychometric_utils.R.
• Response patterns move α, not ω. The paper argues that when items are answered all 1s or all 5s, α can be deflated, while ω stays high because it accounts for item difficulty. This is the authors' explanation, not a tested result app. O. Check item-level response patterns before reading a low α as a reliability failure.
• Base models fail. Base models are not usable as personality instruments in this study, and convergent and discriminant checks fail for all of them. Do not treat the instruction-tuned results as transferring to base models §2.2.2.
• Prompt-set choices matter. The shaping studies used one default preamble and one postamble, to save compute. The authors say scores were robust to preamble and postamble variation in the main construct study, but the shaping results are not re-tested under those variations app. K.
• Quantisation and API drift. PaLM models were quantised, the GPT endpoints may be quantised by the provider, and GPT results depend on dated snapshots app. D. Rerunning with other snapshots may change results.
• Two real-world designs. Flan-PaLM 540B used 100 updates per prompt; other models used 20 updates with 25 repeats. The two designs are not the same sample, so compare the per-model results with that in mind app. M.
• Unstated decoding. Temperature and token limits for the status updates are not stated in the paper. Code defaults differ from each other [code: inference_scripts/generate_status_updates.py#L232-L233] [code: inference_scripts/run_hf_inference.py#L317-L318].
• External API for §4. AMS is an external service. The §4 results cannot be reproduced without access to it.
• Human baseline is not like-for-like. The 0.38 human baseline comes from earlier work on different data §4.2.
• Shaping is shown at the extremes. The all-traits study tests only the two extreme levels, and it uses a fixed set of 32 profiles. Intermediate levels were only tested in the single-trait study §3.1.
• Counting caveat. Section, appendix and table numbers here were counted from the LaTeX source, not from a compiled PDF. Check them against the PDF if exact numbering matters.

Lineage

• Builds on: psychometric measurement of humans (IPIP-NEO, BFI) and the lexical account of personality, which the trait adjectives come from §2 §3.1.
• Compared against: earlier informal questionnaire studies of GPT-3 and other models §2.
• Related concepts: psychometric-scale-validation, llm-evaluation, prompt-engineering, instruction-tuning.