ChatGPT: Jack of all trades, master of none

In plain words

Chatbots are often judged by how fluent their answers sound. This paper asks a plainer question: when you give ChatGPT ordinary text-analysis jobs, such as spotting spam, judging the sentiment of a post, or working out which emotions a sentence conveys, how close does it come to systems built for each job? The authors turned 25 datasets, most of them public, into questions, asked ChatGPT, and scored its answers against the human labels. ChatGPT is a generalist that never beats the best specialised systems here. It does worst on the most subjective tasks, especially emotion. They also tried a few tricks: giving the chatbot labelled examples from one annotator so it could imitate that person, changing the wording of prompts, and checking whether earlier messages in the same chat affect later answers. The numbers describe one chatbot at one time, not a fixed property of the product.

TL;DR

• Evaluates ChatGPT on 25 analytical NLP tasks, comparing its answers with labels and with the best reference system for each task §1 §4 §5.
• Averaged over all 25 tasks, ChatGPT scores 56.51 against 73.71 for the reference, a loss of 25.50% tab. 2.
• The loss is largest on emotion tasks (52.59% on average) and smallest on semantic tasks (12.90%) tab. 2.
• Scores are author-reported. The main runs used an unofficial API client, answers were extracted by hand, and the ChatGPT version and sampling settings are not stated §5.2 §5.3.

Problem

Early ChatGPT evaluations covered one task at a time, were often manual, and used small samples. Benchmark datasets rarely appeared §2. The authors ask whether ChatGPT performs as well as the best recent models on typical analytical NLP tasks, which are mostly classification §3 §4. Many of these tasks are subjective, even for people, so the ground truth is unclear §1. Whether ChatGPT was trained on each dataset is unknown. The authors rate the likelihood by hand §4 tab. 1.

Method

The study runs in three stages: build prompts from each dataset, query the chatbot and extract its answers, then score those answers against a reference §5 fig. 3.

1. Select tasks and samples. Twenty-five tasks, 23 in English and 2 in Polish §4. Most datasets are public; ClarinEmo is unpublished §4. Where a dataset is too large for the query budget, the authors draw a random subset from its test or dev split. Subset sizes are in tab. 1.
2. Build a prompt per item. Each prompt has three parts: a task instruction, the input text, and the allowed labels. The answer is requested as a Python list containing the chosen label, or as a number, emoji or JSON for some tasks §5.1 §7.1.
3. Personalized prompts. For the "Per" tasks and the GoEmoPer variants, the prompt first lists up to three labelled texts from the same annotator, then asks for that annotator's label on a new text §6.3 [app. "Example prompts", AggressionPer]. The selection rule is not stated in the paper; see §Paper vs code.
4. Query. Prompts go through an unofficial API client (PyGPT), using up to 20 accounts in parallel. A new conversation is not forced for each prompt, so earlier messages can remain in context §5.1 §5.3.
5. Extract answers. Raw answers are parsed by hand. Typos in label names are mapped to the intended label. Answers outside the label set become "none" and are excluded from scoring §5.2.
6. Score. F1 macro for most tasks, accuracy for CoLa, WNLI, WordContext and MathQA. Only valid, post-processed cases count §5.4 tab. 1.
7. Reference. Where possible, the authors rerun the best available model (marked "Our" in tab. 2) using its source code. Otherwise they take the score reported in the source paper ("Paper") §5.3.
8. Derived measures. Loss, difficulty and gain, defined below §5.4 §6.2 §6.3.

Further analyses: context and conversation effects on SQuAD §6.4, availability of datasets for training §6.5, prompt patterns §6.6, GPT-4 on five tasks §6.7, expert review of disagreements §7.2, and ChatGPT's own explanations §7.3. Limitations are listed in §8.

Task profile (the paper's counts, per task):

Cat.: P = pragmatic, S = semantic. Test = size of the test or dev split; Used = prompts sent; None = answers outside the label set; N = cases scored tab. 1.

Prompt schema as the paper shows it for Aggression §5.1:

Instruction: Which one of the attributes "aggressive" or "non-aggressive"
             describes a given text? Write the answer as a Python list.
Text:        <input text>
ChatGPT answer (raw):       ["non-aggressive"]
Extracted answer:           non-aggressive

Evaluation loop for one task, as the paper describes it:

# Score one task with ChatGPT  [§4] [§5.1] [§5.2] [§5.4]
# Input:  dataset (texts, gold labels), label set, instruction
# Output: F1 macro, or accuracy for four tasks, over valid answers

def evaluate_task(dataset, label_set, instruction, n_used):
    # Draw a random subset of the test or dev split  [§4]
    subset = random_sample(dataset, size=n_used)
    # One prompt per item: instruction, text, allowed labels  [§5.1]
    prompts = [
        build_prompt(instruction, item.text, label_set)
        for item in subset
    ]
    # Send each prompt and keep the raw answer as returned  [§5.3]
    raw_answers = [chat_model(prompt) for prompt in prompts]
    # Map raw answers to labels; typos are fixed by hand  [§5.2]
    labels = [extract_label(answer, label_set) for answer in raw_answers]
    # Labels outside the set become "none" and are not scored  [§5.2]
    valid = [
        (label, item.gold)
        for label, item in zip(labels, subset)
        if label != "none"
    ]
    return score(valid)  # [§5.4]

Personalized prompt, as the paper describes it. The selection rule is not stated; the code is discussed in §Paper vs code:

# Build a personalized in-context prompt  [§6.3]
# Input:  annotator_items (texts with that annotator's labels), target text
# Output: prompt string

def personalized_prompt(annotator_items, target_text):
    # Pick up to three labelled texts from the same annotator  [§6.3]
    examples = select_examples(annotator_items, max_count=3)
    # Show each example as text with the annotator's label  [§6.3]
    context = "\n".join(
        "Text: " + example.text + "; Attribute: " + example.label
        for example in examples
    )
    # Ask for the label of the new text in the same format  [§5.1]
    return context + ask_for_label(target_text)

Key equations. The paper does not number them; they are cited by section.

F1_{macro} = \frac{1}{Q} \sum_{i=1}^{Q} \frac{2\, p_i\, r_i}{p_i + r_i} §5.4

Here Q is the number of labels, and p_i and r_i are the precision and recall for label i §5.4.

\mathrm{Loss} = \frac{100\% \cdot (\mathrm{SOTA} - \mathrm{ChatGPT})}{\mathrm{SOTA}} §5.4

\mathrm{Difficulty} = 100\% - \mathrm{SOTA} §6.2

\mathrm{Gain} = \frac{100\% \cdot (\mathrm{Per} - \mathrm{NonPer})}{100\% - \mathrm{NonPer}} §5.4

Gain compares a personalized in-context run (Per) with the non-personalized reference (NonPer), both as F1 values §5.4 fig. 9.

Hyperparameters

The paper gives no model version and no sampling settings. Items marked "not stated" should be taken as unknown, not as defaults.

Results

Author-reported. ChatGPT is run through the unofficial client; the reference is the authors' rerun ("Our") or the value from the source paper ("Paper"). Scores are percentages.

Table 2 (all 25 tasks). Difficulty = 100 − reference; Loss from the formula above tab. 2:

Asterisk = emotion task. Summary rows, as the paper reports them tab. 2:

Other reported results:

• Difficulty and loss. Pearson correlation between difficulty and loss is 0.46 across the 25 tasks. The authors call the sample small §6.2.

• Personalization §6.3 fig. 9. The paper reports loss gaps for AggressionPer and UnhealthyPer and a gain for GoEmoPer3 over GoEmoPer0 in the text. These numbers do not match tab. 2; see §Pitfalls.

• Context, SQuAD v2 §6.4 tab. 3:

  Setup	Accuracy	F1	Unanswerable detected	Not detected
  Original set	56.50	69.21	76 (25.33%)	224 (74.67%)
  One week later	55.40	68.72	64 (21.33%)	236 (78.67%)
  New order (unanswerable first)	57.00	69.76	74 (24.67%)	226 (75.33%)
  Separate conversations	53.60	67.23	60 (20.00%)	240 (80.00%)

  The authors say these differences are not significant and that repeated launches vary.

• Prompt patterns §6.6 tab. 4: initial ChatGPT average 59.4, pattern average 56.5 (−2.9). Table rows point to appendix chats by label:

  ID	Task	Chat (label in source)	Initial (a)	Pattern (b)	Difference
  8	TextEntail	eng7	88.1	77.6	−10.5
  8	TextEntail	eng8	88.1	81.2	−6.9
  8	TextEntail	eng9	88.1	77.6	−10.5
  8	TextEntail	eng10	88.1	75.1	−13.0
  9	WNLI	eng11	81.7	74.2	−7.5
  9	WNLI	eng12	81.7	77.5	−4.2
  9	WNLI	eng13	81.7	76.1	−5.6
  13	GoEmo	eng0	25.6	21.8	−3.8
  13	GoEmo	eng1	25.6	26.4	+0.8
  13	GoEmo	eng2	25.6	23.7	−1.9
  13	GoEmo	eng3	25.6	24.6	−1.0
  20	PolEmo	eng4	44.1	38.6	−5.5
  20	PolEmo	eng5	44.1	57.8	+13.7
  20	PolEmo	eng6	44.1	58.9	+14.8

• GPT-4 on five tasks §6.7 tab. 5. Both models use the same prompts. "All tasks" is the average over these five tasks only.

  ID	Task	ChatGPT (a)	GPT-4 (b)	Difference	Reference (c)	ChatGPT loss	GPT-4 loss
  8	TextEntail	88.1	91.3	3.2	92.1	4.3	0.9
  9	WNLI	81.7	91.6	9.9	97.9	16.5	6.4
  10	SQuAD	69.2	76.3	7.1	90.8	23.8	16.0
  13	GoEmo	25.6	23.1	−2.5	52.8	51.6	56.3
  20	PolEmo	44.1	41.0	−3.1	76.4	42.3	46.3
  	All five	61.7 (printed "61,7")	64.7	2.9	82.0	27.7	25.2

• Expert review of label disagreements §7.2 tab. 6. Share of values the experts accepted, for human annotations and for ChatGPT answers:

  Task	Human annotation approved	ChatGPT answer approved
  Aggression	68%	51%
  TweetSent	69%	55%
  GoEmo	61%	73%
  Unhealthy	43%	81%

  A finer split is in tab. 7. For example, for Unhealthy, 57% of cases had only ChatGPT's label accepted, and for Aggression, 48% had only the human label accepted.

Paper vs code

The official repository is CLARIN-PL/chatgpt-evaluation-01-2023 at commit 737e3882f50dff0de63d4fa95c63425afc01d06c, under the MIT license (copyright CLARIN-PL, 2023). The paper links it in §5.1; its README cites this paper. The repository holds prompt-generation notebooks, the ChatGPT and reference runs, an interrogation client (interrogation-api/), and the results sheet ChatGPT Evaluation v2.0.xlsx. We did not rerun any query or recompute any score.

Discrepancies:

• Personalized context selection, AggressionPer. The paper's title and §6.3 describe "random" contextual few-shot personalization. The notebook takes, for each annotator, the three texts with the highest inter-annotator standard deviation, sorted in descending order, from the annotator's non-selected items 02_AggressionPer.ipynb cell-9. This is deterministic, not random. The notebook is one JSON line, so only the cell is cited.
• Personalized context selection, GoEmoPer. Here the context is random: up to three other texts from the same rater, drawn with pandas sample and no random_state in that cell 14_15_16_17_GoEmoPer_0_1_2_3.ipynb. Before that, a seeded draw (seed 7) keeps up to 15 items per rater 14_15_16_17_GoEmoPer_0_1_2_3.ipynb.
• Prompt formatting, AggressionPer. The notebook joins context items with "; Attribute:" on one line each. The paper's example puts "Attribute:" on its own line app. "Example prompts", AggressionPer. This is minor but matters for exact replication.
• Access. The paper says the main runs used the PyGPT client §5.3. The repository's client reads a session token from a browser cookie file and waits 600 s after a rate-limit status interrogate.py interrogate.py. This is consistent with the paper but shows that the runs depended on a live web session.
• Model and settings. Neither the model version nor any sampling parameter appears in the repository. The grep covered src/pygpt.py and both READMEs.

Pitfalls

• Unstated model and settings. The version of ChatGPT, the query dates, and the sampling settings are not given. The authors observed non-deterministic answers across launches §6.4. Treat the scores as a snapshot, not as a property of a named model.
• "None" answers are excluded. Answers outside the label set, and policy refusals, are removed before scoring §5.2 §8 items 3, 4 and 9. The scores measure accuracy on valid answers, not coverage. Per-task None counts range from 0 to 28 tab. 1.
• Manual extraction and label mapping. Typos were mapped to labels "assuming the intended answer" §5.2. No written rules are given. Check the raw answers in the repository's spreadsheet before relying on any mapping. Post-processing affects more than 15% of cases for Aggression, SQuAD, Unhealthy and ReAding §5.2.
• Subsets, no intervals. Most tasks are scored on random subsets §4. The paper gives standard deviations across tasks but no confidence intervals per task tab. 2. Differences of a few points may be noise.
• Mixed references. "Our" rows use the authors' rerun; "Paper" rows use numbers from other setups and splits §5.3 tab. 2. The authors note the setups often differ. Loss values mix both kinds.
• Mixed metrics. Accuracy is used for CoLa, WNLI, WordContext and MathQA; F1 macro otherwise §5.4. The overall averages combine both.
• Loss and difficulty are linked. Difficulty is 100 minus the reference score §6.2, so low-scoring references produce high difficulty and large losses partly by construction. The correlation of 0.46 is from 25 tasks, and the authors call it preliminary §6.2.
• Availability is a judgement. The 0–3 availability ratings are the authors' estimates §4 tab. 1. The "Trained" column records ChatGPT's own claim §6.5, which the authors did not verify.
• Personalization numbers in the text do not match Table 2. The text gives a gain of 4.08 pp for GoEmoPer3 over GoEmoPer0 §6.3. Table 2 gives 23.41 against 23.74, a difference of −0.33 pp tab. 2. The 3.25 and 12.55 pp loss gaps for AggressionPer and UnhealthyPer §6.3 match no difference in tab. 2, where the gaps against the reference are 8.46 and 16.90 pp. Figure 9 carries the gain values but no numbers in the text fig. 9. More context did not help monotonically: GoEmoPer1 (19.00) and GoEmoPer2 (20.34) score below GoEmoPer0 (23.74) tab. 2.
• Context effects are not significant. The authors say the differences in the SQuAD context test are not significant §6.4.
• Prompt wording moves results a lot. One prompt per task is used for the main numbers §5.1. The prompt was tuned by hand on a small validation sample §7.1. Changing the wording moved PolEmo by up to +14.8 pp and TextEntail by up to −13.0 pp tab. 4. The text cites "patterns 76 and 77"; the table lists chat labels eng5 and eng6, and we could not map the numbers without rendering.
• Access limits and account counts. Section 5.3 says up to 20 accounts; §8 item 6 says 13. Throughput was about 2–3k prompts per day at the end §8 item 6.
• Table artifacts. The GPT-4 average is printed "61,7" tab. 5. The "All tasks" rows in tab. 4 and tab. 5 average different subsets, so they cannot be compared with the 25-task average of 56.51 tab. 2.
• Expert review is informal. The two experts had no written guidelines and no inter-annotator agreement is reported §7.2. The approval shares tab. 6 are not reproducible measures.
• GPT-4 is a small sample. Only five tasks, limited by a prompt quota §6.7.
• Version. This card follows arXiv v4 (updated 2023-06-09), the version in our metadata.

Lineage

• Builds on: InstructGPT, whose reinforcement learning from human feedback motivates the study §1 (not carded; named only). The authors' earlier work on personalized NLP is cited for the personalization idea §6.3 (not carded; named only).
• Compared against: the best available reference model for each task tab. 2, and GPT-4 on five tasks tab. 5.