Paper · 2025-05-14 · cs.CL
When Large Language Models are More Persuasive Than Incentivized Humans, and Why
In a real-time chat quiz, Claude 3.5 Sonnet out-persuaded incentivized human persuaders toward both correct and wrong answers. Accuracy rose under truthful persuasion and fell under deceptive persuasion. The LLM edge shrank over repeated questions. In a follow-up, DeepSeek v3 beat humans only when deceptive. Author-reported.
LLM evaluationHuman evaluationLLM ethicspersuasiondeceptionhuman-subject-studymisinformationllm-chat
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Chatbots can talk people into things. This study asks a sharper question: when a chatbot tries to steer someone toward an answer, does it do better than a person who is paid to do the same job? Online participants answered a short quiz. Before each answer, they chatted with a partner, and the partner was either a paid human or a language model. The partner was told to push the quiz taker toward either the right answer or a wrong one.
In the main study, the language model beat the paid humans on both kinds of push. When it pushed toward right answers, people got more of them right. When it pushed toward wrong answers, people got more of them wrong. The model's advantage got smaller as the quiz went on, while the humans' effect stayed flat. The authors look at how the model wrote its messages and find that it sounded more certain, using strong words such as "completely" more often than the humans did. That confident style may be part of why it worked, but the authors say this is only a partial explanation.
TL;DR
Claude 3.5 Sonnet was a more effective persuader than incentivized human persuaders in a 10-question quiz, both when pushing toward correct answers and toward wrong ones §3.1. Accuracy went up under truthful persuasion and down under deceptive persuasion, for Claude and for humans tab. 2. In a follow-up with DeepSeek v3, the LLM advantage held only for deceptive persuasion §3.2. The advantage weakened over repeated questions for LLMs but not humans §3.3. Everything is author-reported, and the study measures one quiz task with one payment scheme.
Problem
Earlier studies mostly compared LLMs with unmotivated human persuaders, measured stated attitudes instead of answers, or used one-shot messages §1.2. This study tests whether LLMs outperform humans who have a financial reason to persuade, in a live chat, on questions with verifiable answers, and whether the LLM does better when pushing toward truth or toward falsehood §1.2 §1.3. Five pre-registered questions guide the analysis: RQ1 overall persuasiveness, RQ2 truthful compliance, RQ3 deceptive compliance, RQ4 accuracy under truthful persuasion, and RQ5 accuracy under deceptive persuasion §1.3.
Method
Study 1 (pre-registered). Prolific participants were randomly placed into one of two roles, quiz taker or persuader §2.1 fig. 2. Quiz takers were then assigned to one of three conditions: a solo control with no partner (20%), a human persuader (40%), or an LLM persuader (40%) §2.1 fig. 2. The LLM persuader was Claude 3.5 Sonnet §2.1. The quiz had 10 binary questions drawn from three sets: trivia, illusion and forecasting app. G fig. 2. Each question was assigned a direction at random (within subject): "positive" meant push toward the correct answer, and "negative" meant push toward the wrong one §2.1 app. A. Quiz takers were told that their partner could be another human or an AI, but not which one p. 7. Each question was answered on a two-option choice plus a 0–100 confidence rating §2.1 app. G. The study was pre-registered p. 5, footnote 1.
Study 2 (follow-up). DeepSeek v3 replaced Claude as the persuader. Participants were assigned to the LLM condition with probability 66% or to solo control with 33% p. 6. There was no human arm in Study 2. The forecasting set was replaced by 18 financial and conspiracy questions §3.2 app. G. Human comparisons use only the illusion and trivia questions and draw on Study 1 participants p. 10, footnote 2 §3.2.
Persuader instructions. The LLM persuaders received a role-play system prompt, reproduced in app. A. It tells the model to push the quiz taker toward the assigned answer, to never disclose its role, to write short casual messages, and to stay in character. Each question is passed in a fixed template with the question type, the answer options, the private correct answer or forecast trend, and the direction app. A. Human persuaders were given the correct answer, or the forecast trend, as the LLM persuaders were, and were paid for successful persuasion app. G app. G.2. Two-option answers were set so that the wrong answer was always fixed for each question, so both human and LLM persuaders steered toward the same target app. G §2.1.
Scoring.
# Per quiz taker, over all non-forecast questions [app. G.3]
# Input: answers, correct flags, directions, forecast flags
# Output: accuracy and compliance rates
def score_participant(answers, correct, directions, is_forecast):
# Accuracy: share of non-forecast questions answered correctly [app. G.3]
scored = [i for i in range(10) if not is_forecast[i]]
accuracy = mean(answers[i] == correct[i] for i in scored)
# Compliance: answer matches the persuader's intended direction [app. G.3]
# For forecasts, "correct" means the two-week trend [app. G.3]
compliance = mean(complied(answers[i], directions[i]) for i in range(10))
return accuracy, complianceMechanism analyses. The study looks at word counts, readability, emotional tone, and three markers of confidence: boosters, maximizers and hedges, using word lists in fig. S2 §3.4. It also fits mediation models in lavaan with 1,000 bootstrap samples §3.4. Linguistic features are reported as exploratory §3.4 p. 16.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Persuader model, Study 1 | Claude 3.5 Sonnet, endpoint claude-3-5-sonnet-20241022 | §2.1 | Endpoint ID given in the text |
| Persuader model, Study 2 | DeepSeek v3 | §3.2 | Version, endpoint and provider not stated |
| Sampling settings (temperature, top-p, max tokens) | not stated | app. A | Only prompts are given |
| Condition probabilities, Study 1 | solo 20%, human 40%, LLM 40% | fig. 2 §2.1 | Random assignment |
| Condition probabilities, Study 2 | LLM 66%, solo 33% | p. 6 | Random assignment |
| Questions per quiz | 10 | §2.1 app. G | At least 3 from each set fig. 2 |
| Answer options | 2 per question | §2.1 app. G | Binary; wrong answer fixed per question |
| Question direction | positive or negative, random per question | §2.1 | Within subject |
| Confidence rating | 0–100 | §2.1 | Given with each answer |
| Minimum messages | at least 2 per question for each side | app. G.2 | Who opens is randomised |
| Time to answer | up to 3 min per question | app. G.2 | Solo condition advances after 20 s |
| Minimum wait, non-solo conditions | 2 min | app. G.2 | Per question |
| Sample, Study 1 | 1,242 recruited; 888 quiz takers; 354 in each persuader condition | fig. 2 app. G.2 | Solo control: 180 quiz takers |
| Sample, Study 2 | 559 recruited; 374 LLM, 185 solo | app. G.1 | Pay as in Study 1 |
| Power plan, Study 1 | 1,050 total (750 quiz takers, 300 persuaders) for d ≈ 0.27 at 90% power | app. G.1 app. C | Pilot data with 35 participants |
| Base pay | GBP 5 per participant | app. G.2 | About GBP 10.12/hour, per the authors |
| Bonus pay | GBP 10 to the top 50 persuaders, 50 accurate treatment quiz takers, 25 accurate solo quiz takers | app. G.2 | Study 1 only |
| Forecast trend | two-week historical trend given to persuaders | app. G app. A | 14 of 18 forecasts matched the trend |
| Mediation bootstrap | 1,000 samples | §3.4 | lavaan in R |
Results
Author-reported. Setup: US participants on Prolific; Claude 3.5 Sonnet and DeepSeek v3 as persuaders; human persuaders paid for success; solo quiz as control §2.1 app. G.2. Compliance is the share of answers that follow the persuader's direction. Accuracy is the share of correct answers.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| Study 1, overall | compliance (%) | human 59.91 (SD 19.44) | Claude 67.52 (SD 20.21); diff +7.61 [4.66, 10.56], p < .001 | tab. 1 §3.1 |
| Study 1, truthful | compliance (%) | human 85.13 | Claude 88.61; diff +3.48 [0.82, 6.14], p = .010 | tab. 1 §3.1 |
| Study 1, deceptive | compliance (%) | human 35.36 | Claude 45.67; diff +10.31 [5.87, 14.76], p < .001 | tab. 1 §3.1 |
| Study 1, truthful | accuracy change vs solo (pp) | human +7.8 (SE 2.0), accuracy 78.0 | Claude +12.2 (SE 2.0), accuracy 82.4; control 70.2 | tab. 2 |
| Study 1, deceptive | accuracy change vs solo (pp) | human −7.8 (SE 2.6), accuracy 62.4 | Claude −15.1 (SE 2.6), accuracy 55.1 | tab. 2 |
| Study 2, overall | compliance (%) | human 59.34 | DeepSeek 62.55; p = .073 (n.s.) | §3.2 |
| Study 2, truthful | compliance (%) | human 89.75 | DeepSeek 91.89; p = .161 (n.s.) | §3.2 |
| Study 2, deceptive | compliance (%) | human 28.23 | DeepSeek 34.73; p = .013 | §3.2 |
| Study 2, truthful | accuracy (%) | solo 75.3 | DeepSeek 83.6 (+8.3 pp, p < .001) | §3.2 |
| Study 2, deceptive | accuracy (%) | solo 75.3 | DeepSeek 57.8 (−17.5 pp, p < .001) | §3.2 |
Other reported results, all author-reported:
- Maximizer density (per 100 words) was 0.09 for humans, 0.23 for Claude and 0.47 for DeepSeek fig. S2. Hedging density was lower for both LLMs than for humans fig. S2.
- Participants paired with Claude who answered later questions complied less. Compliance with Claude fell about 1.0 point per question, and with DeepSeek about 0.8 point per question. Human compliance was flat §3.3 fig. 6.
- Linguistic complexity: AI messages were longer and harder to read than human messages by every measure except lexical diversity, where humans were higher tab. 3.
- Maximizer density (words such as "completely" or "absolutely") explained part of Claude's compliance advantage in a single-mediator model (indirect effect 1.53, 95% CI [0.77, 2.77]) and remained significant in the parallel model (1.41, 95% CI [0.72, 2.69]). Hedging did not mediate §3.4.
- Robustness: the overall and deceptive findings held after removing extreme questions and after removing forecasts. The truthful compliance gap for Claude was no longer significant after removing very easy and very hard questions (p = .113) app. F tab. S2.
Paper vs code
No official model code. The experiments use hosted LLM endpoints and the prompts in app. A. The paper's Data Availability statement names an OSF project, osf.io/mtyqz, for code and related data p. 23. That project is public, lists analysis scripts (Python and R) and data folders, and has no license on OSF p. 23. We did not read these files: the download links redirected to signed storage that the fetch tool could not open. Nothing from them is quoted or cited by line, and their analyses have not been checked against the reported numbers. The pre-registration is a separate OSF link p. 5, footnote 1.
Known textual inconsistencies between paper and its own numbers:
- The robustness table is cited as "Table 4" in the text app. F p. 43, but the PDF labels it Table S2 tab. S2.
- Appendix G.2 refers to "Section 2.3" for incentive details, and the PDF has no Section 2.3; the incentive rules are in §2.1 and app. G.2 p. 47.
- Table 2 gives n = 353 for human truthful and n = 354 for Claude truthful, while Figure 2 gives N = 354 for each persuader condition tab. 2 fig. 2.
Pitfalls
- Study 2 is not a fair human comparison. Humans in Study 2 were not recruited; the comparison uses Study 1 humans and only illusion and trivia items, and the two studies were not randomised together p. 10, footnote 2 §3.2.
- Forecast questions have no answer at the time. Persuaders were told a two-week historical trend and "correct" meant matching that trend. The authors checked that 14 of 18 forecasts resolved in the trend direction app. G.3, footnote 6 app. G.
- Deception is built into the prompt. Persuaders were told to lie and never disclose their role, and the quiz takers were told the partner might be a human or an AI app. A p. 7. Ethics approval and debriefing are described in app. G.1. Any replication must follow the same consent and debrief rules.
- Model settings are missing. No temperature, top-p or token limits are given, and the DeepSeek version and endpoint are not stated app. A §3.2. The Claude endpoint is a dated snapshot, so re-running with another version may not match §2.1.
- Mediation is exploratory. The mediation and linguistic analyses were not pre-registered, and the authors say they should be read with caution §3.4 p. 16.
- LLM-coded measure. The repetition counts in tab. S1 were coded by Claude 4.6 Sonnet, and two responses were malformed. Some analysis code was written with LLM help and reviewed by hand tab. S1 p. 23.
- Ceiling and floor. Control accuracy is about 70%, which limits how much truthful persuasion can add and leaves more room for deceptive effects §4.2 tab. 2.
- Order effects. The LLM advantage narrows across the 10 questions §3.3 fig. 6. The headline effects depend on the 10-question design, which the authors chose for power app. C.
- Source identity. 51% of quiz takers in the human condition wrongly thought their partner was an AI, while 91% in the Claude condition correctly identified an AI. Belief about the source is therefore not a clean check on the manipulation p. 44 fig. S1.
- Costs that the abstract hides. Results depend on paid bonuses for both sides and on a financial stake for quiz takers, which the authors say may reduce ecological validity §4.2 app. G.2.
Lineage
- Builds on: the Empirica platform for the experiment §2.1. The rule that participants may not use AI or web-search tools follows Veselovsky et al. 2025 §2.1.
- Compared against: earlier LLM and human persuasion studies, including Bai et al. 2025, Salvi et al. 2025, Karinshak et al. 2023, Costello et al. 2024 and Hackenburg et al. 2025 §1.1 §1.2.
- Superseded by: none stated.
Source and license
Adapted from “When Large Language Models are More Persuasive Than Incentivized Humans, and Why” by Jiacheng Liu, Francesco Salvi, Philipp Schoenegger, Xiaoli Nan, Ramit Debnath, Barbara Fasolo, Evelina Leivada, Gabriel Recchia, Fritz Günther, Ali Zarifhonarvar, Joe Kwon, Zahoor Ul Islam, Marco Dehnert, Daryl Y. H. Lee, Madeline G. Reinecke, David G. Kamper, Mert Kobaş, Adam Sandford, Jonas Kgomo, Luke Hewitt, Shreya Kapoor, Kerem Oktar, Eyup Engin Kucuk, Bo Feng, Cameron R. Jones, Izzy Gainsburg, Sebastian Olschewski, Nora Heinzelmann, Francisco Cruz, Ben M. Tappin, Tao Ma, Peter S. Park, Rayan Onyonka, Arthur Hjorth, Peter Slattery, Qingcheng Zeng, Lennart Finke, Igor Grossmann, Alessandro Salatiello, Ezra Karger (arXiv:2505.09662), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.