Art or Artifice? Large Language Models and the False Promise of Creativity

In plain words

Writing assistants are getting good at producing stories, but it is hard to say whether a story is creative. Creativity is usually measured by watching a person think, which is not possible for a finished story or a language model. The authors take a classic creativity test and turn it into a checklist for the finished text: fourteen yes-or-no questions, grouped under four headings.

Ten experts in creative writing used that checklist on 48 short stories, some written by professional authors and some written by AI models. The human stories passed most of the questions. The AI stories passed far fewer. When the authors asked AI models to do the same checklist, their answers did not agree with the human writers.

The checklist is a way to compare stories in aggregate. No single question decides whether a story is creative.

TL;DR

The paper defines the Torrance Test of Creative Writing (TTCW): 14 binary tests for short fiction, grouped under fluency, flexibility, originality and elaboration §4.2. Ten expert writers scored 48 stories, 12 from The New Yorker and 36 generated by three LLMs §5.1, §5.3. Stories by the professional authors passed 84.7% of tests, against 8.7% to 30.0% for the LLM stories tab. 5. The LLMs used as judges did not agree with the expert majority, with average Cohen's kappa between -0.006 and 0.035 tab. 9. The benchmark data and prompts are released README.md.

Problem

Evaluating the creativity of a piece of writing is hard, and the original Torrance tests measure creativity as a process, not as a product §1. Their four dimensions transfer across domains, but their direct application may not §3, DP1. Process-based evaluation also fails for finished artifacts and for black-box models whose process cannot be observed §3, DP2. The paper assumes short fictional stories, a panel of qualified experts judging with the Consensual Assessment Technique, and binary judgements with written rationales §1, §3, DP3.

Method

The work has three parts: (1) design the tests with a formative study of experts §4; (2) have experts administer the tests to a 48-story benchmark §5; (3) ask LLMs to administer the same tests §6.

Design principles §3:

• DP1, grounded in the Torrance dimensions: fluency, flexibility, originality and elaboration §3, DP1.
• DP2, artifact-centric: evaluate the finished story, not the writer's process §3, DP2.
• DP3, binary yes/no per test, each with a written rationale of one to three sentences §3, DP3.
• DP4, additive: the tests are independent, none is a prerequisite, and the score is the number of tests passed §3, DP4.

Formative study §4: eight recruited writers (creative-writing professors, MFA students, a young-adult novelist and a screenwriter) gave 126 measures across the four dimensions. Three authors coded them inductively into 14 groups, and one representative question was kept or converted to yes/no form with GPT-4 §4.1. The 14 tests in §4.2 are:

Source: §4.2; the expert-to-test mapping is in tab. 2.

Benchmark construction §5.1:

1. Collect 12 New Yorker short stories with diverse authors and plots, published between August 2020 and May 2023 and 1,000 to 2,400 words long §5.1.
2. For each, have GPT-4 write a one-sentence plot summary, then check it by hand §5.1, fig. 2.
3. Prompt three LLMs (GPT-3.5, GPT-4, Claude v1.3) to write one story per plot, with a length target equal to the human story §5.1, fig. 2.
4. Because the models wrote 20 to 50% short, expand each story iteratively until the word-count gap is under 200 words; all models converged within 20 iterations §5.1, app. "NewYorker Data for evaluation".
5. Result: 12 groups of four stories, one human and three LLM, 48 stories in all §5.1. The average LLM story is about 1,400 words §1.

Expert protocol §5.2:

1. Shuffle and anonymise the four stories in a group §5.2.
2. For each story, read it in full and answer the 14 tests with Yes or No and a written reason §5.2, DP3.
3. Afterwards, rank the four stories by preference and guess each one's origin: experienced writer, amateur, or AI §5.2, fig. 4.

Each expert received an expanded definition of each test, produced by GPT-4 and checked by three domain experts. The expanded text and the final instruction are in tab. 3 (row 2 is the expanded measure, row 3 the human instruction).

LLM-as-assessor protocol §6:

1. Use the same 48 stories and 14 tests, with GPT-3.5, GPT-4 and Claude §6.
2. Each prompt has four parts: the story, the expanded test context, the test question, and an instruction to reason step by step before answering Yes or No §6.
3. Prompts are zero-shot. Few-shot prompts were tried on a few tests with no significant gain §6.1.
4. Compare each LLM's Yes/No with the majority vote of the three experts, using Cohen's kappa §6.

Pseudocode. The paper gives no numbered algorithms or equations. The scoring rule and the judge-prompt assembly are summarised below. The first block follows the released benchmark code Evaluating_LLM.ipynb.

# Build the context shown to the judge for one test  [§6] [tab. 3]
# Input:  full_prompt (str), the expanded test text
# Output: context (str), the test text without its last two lines
def expanded_context(full_prompt):
    # Drop the trailing question line and the step-by-step line
    # [code: Art_or_Artifice/Evaluating_LLM.ipynb#L40-L46]
    lines = non_empty_lines(full_prompt)
    return "\n".join(lines[:-2])

# Fill the shared judge template for one (story, test) pair  [§6]
# Template: Art_or_Artifice/prompts/with_background.txt
# [code: Art_or_Artifice/prompts/with_background.txt#L1-L20]
def build_judge_prompt(story, test, template):
    # Replace the three placeholders with the story, context and question
    prompt = template.replace("[STORY]", story)
    prompt = prompt.replace("[BACKGROUND]", test.context)
    return prompt.replace("[QUESTION]", test.question)

# Ground truth per (story, test): majority of 3 expert labels  [§5.3] [§6]
# Input:  verdicts (list of 3 str, each "Yes" or "No")
# Output: label (str)
def majority_label(verdicts):
    # Binary labels with three votes cannot tie
    # [code: Art_or_Artifice/Evaluating_LLM.ipynb#L64]
    return most_common(verdicts)

# Story score: number of the 14 tests the story passes  [§3, DP4] [§5.4.2]
# Input:  passes (list of 14 bool)
# Output: score (int, 0 to 14)
def story_score(passes):
    # Tests are additive; no single test is decisive  [§3, DP4]
    return sum(passes)

Hyperparameters

The paper reports no training hyperparameters; it is a human and LLM evaluation study. The values below are the experimental settings that the paper states.

Results

Author-reported. Setup: 48 stories (12 human, 36 LLM), 10 expert annotators, 14 binary tests, 3 annotators per story and test §5.1, §5.3. Passing rate is the share of Yes votes §5.4.1.

Human judgements, pass rate per test (%) and expert agreement tab. 5:

Headline numbers from the text §5.4.1, §5.4.2:

• Human stories pass 84.7% of tests on average, about 11.9 of 14 per story. No test reaches 100%.
• LLM stories pass between 8.7% (GPT-3.5) and 30.0% (Claude v1.3) of tests, roughly a third to a tenth of the human rate. The authors also describe this as "three to ten times less likely" §1.
• Expert agreement on individual tests ranges from 0.27 to 0.66 (Fleiss kappa 0.41 average), which the authors call moderate.
• At the aggregate level, the number of tests passed correlates across experts with Pearson 0.69, which the authors call strong.
• In the ranking task, the human story was ranked most preferred 89% of the time §5.4.3.

LLM judges, Cohen's kappa against the expert majority tab. 9:

The authors conclude that none of the LLMs correlate positively with experts. GPT-4 is the only model above 0.2, on two of the 14 tests §6.1. Few-shot prompts gave no significant gain on the tests tried §6.1.

Other reported findings: the expert explanations point to recurring themes for each test, for example weak narrative endings and clumsy metaphors in AI stories §5.5, tab. 7, tab. 8. Experts said they tell AI from human writing by creative execution, not by grammar §5.6. The authors' comparative numbers for origin guesses are shown in fig. 4.

Paper vs code

The official repository is salesforce/creativity_eval at commit 3d029879df6878f611363db88cc02d465699bc51, BSD-3-Clause LICENSE. The authors are at Salesforce AI Research, and the first author's public fork (tuhinjubcse/creativity_eval) states it is the code repository for this paper. The code is a data and benchmark release, not the story-generation pipeline.

• Judge temperature and sampling. The paper says experiments used default settings with T = 1.0 §8. The runner gets its model and kwargs from an external model_bank module, imported from an absolute path run_llm_eval.py. Temperature and other settings are therefore not stated in the repository.
• Step-by-step reasoning. The paper says the LLM prompt asks for reasoning step by step before the answer §6. The per-test prompts in tests/ttcw_all_tests.json contain such an instruction. The benchmark builder drops the last two lines of each prompt before building the context Evaluating_LLM.ipynb, and the shipped runner template only asks for an optional explanation after Yes or No with_background.txt, with_background.txt. The released runner therefore does not elicit step-by-step reasoning as the paper describes.
• Ground truth. The majority vote of three experts matches the paper Evaluating_LLM.ipynb §6. No discrepancies found.
• Agreement statistics. The Fleiss kappa, Pearson correlation and the Cohen kappa for LLMs are not computed in the repository. The notebooks contain no kappa or correlation code. Values in the paper cannot be reproduced from the release.
• Benchmark file. data/crea_eval_dataset.json is read by the runner run_llm_eval.py but is not in the repository. It is built by a notebook cell whose write-out is commented out Evaluating_LLM.ipynb.
• Story generation. The code to generate the 36 LLM stories, including the expansion loop, is not in the repository. The prompts are in [app. "NewYorker Data for evaluation"].
• Model set. The paper's judges are GPT-3.5, GPT-4 and Claude v1.3 §6. The README lists later results for GPT-3.5-turbo, GPT-4-Turbo, Gemini Pro and Claude 1.3, 2.0, 2.1 and 3 Opus, updated in March 2024 README.md. Those judgements are not in the paper and are not reported here.
• Dataset release. The 12 New Yorker stories are not included; the release links to them README.md.
• Links in stories. The notebook skips any story whose text contains a link, and asks the user to replace the URL with the full story text Evaluating_LLM.ipynb. A rebuild therefore needs the New Yorker text, which is not in the repository.

Pitfalls

• Ground truth is noisy. Expert agreement is moderate per test (0.27 to 0.66). Only the aggregate count is reproducible at the level the authors claim §5.4.2. Do not read a single test as a decision.
• Additive score. Failing one test does not mean a story is not creative. The paper cautions that the number of tests passed is the measure §3, DP4.
• Plot is shared. LLM stories are generated from the same plot summary as the human story. This isolates prose quality but does not measure plot originality §5.1.
• Length is controlled, not free. Stories were expanded until the length gap was under 200 words §5.1. Replications must match this, or length will confound the results.
• Sampling is default and untuned. T = 1.0 is the paper's choice, and the authors say other settings may change the pass rates §8.
• Judge prompt mismatch. The shipped runner does not ask for step-by-step reasoning (see Paper vs code). Results from the released runner are not directly comparable with the paper's Table 9.
• Small expert pool. Ten writers, three judgements per story, 12 story groups §5.3. Coverage of creative-writing criteria is limited; experts asked for a "voice and diction" test that the paper did not include [app. "Expert Perception on the TTCW tests"].
• Rounding. The introduction says GPT-3.5 passes 9% of tests §1. Table 5 gives 8.7%. Use the table.
• Wording of the correlation claim. The abstract says no LLM positively correlates with experts. Two averages in tab. 9 are slightly positive (0.016 and 0.035), and the per-test values are near zero.
• Ratio is approximate. Dividing the averages in tab. 5 gives about 2.8x (30.0 vs 84.7) to 9.7x (8.7 vs 84.7). The paper rounds this to "three to ten times".
• Cost. About $20 and 30 minutes of expert time per story §6. Human pay was $80 per group of four §5.2.
• Licence of text. The New Yorker stories are copyright material and the release does not include them. Do not reproduce them from memory or from outside sources.

Lineage

• Builds on: the Torrance Tests of Creative Thinking §1, §3, DP1 and the Consensual Assessment Technique §1. Neither has a card yet.
• Uses: chain-of-thought prompting for the LLM judges §6.
• Compared against: LLM-based evaluation of creative output with GPT-4 on short tasks, which the authors say does not transfer to longer stories §6.1.