wikillm

Paper · 2024-06-19 · cs.AI, cs.CL

Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency

Author-run evaluation of GPT-4o on exams, logical reasoning, translation, image classification, captioning, VQA, speech emotion and accent tasks, and MM-Vet, using 20 to 500 items per task. Strongest on exams and deductive reasoning (83.05% on a USMLE Step 1 sample, 85.39% on a CFA Level 1 mock); weakest on CT cancer classes and accent recognition. Author-reported.

LLM evaluationMultimodal language modelBenchmark validityFew-shot learningevaluationbenchmarkmultimodalvisionspeechreasoningtranslation

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

This paper does not propose a new model. It tests GPT-4o, a model from OpenAI that handles text, images and audio, on a broad set of everyday benchmark-style tasks: exam questions, logic puzzles, translation, image recognition, sound and speech classification, and questions about pictures. Each task gets a plain prompt, and the answers are compared with a known correct answer.

The results are mixed. The model does well on exams and on text-based reasoning. It is weak on several image and audio tasks, such as the CT brain scan categories and accent recognition. The samples are small, each task was run once, and some comparison numbers come from earlier papers rather than from the authors' own runs. So the numbers are a snapshot of one model at one time, not a settled ranking.

Treat the card as a list of checks to run yourself before trusting this model in your own product, not as a verdict on the model.

TL;DR

  • An evaluation study, not a method: GPT-4o is tested across language, vision, speech and multimodal tasks, with about 20 to 500 items per task §2–§5.
  • Strong results on exams (83.05% on a USMLE Step 1 sample tab. 1) and on deductive reasoning tab. 6; weak results on CT cancer classes tab. 12 and accent recognition fig. 15.
  • Single runs, small samples, no code, and some baselines copied from other papers. Use the numbers as a starting point, not as targets to match.

Problem

The authors state that GPT-4o's language, vision, speech and multimodal abilities had not been formally evaluated in the literature, and that earlier GPT-4 and GPT-4V evaluations were limited in scope §1, §1.2. They ask for broader benchmarks, error analysis and human judgement §6.2. The study assumes public sample datasets and zero-shot or few-shot prompting, with no training or fine-tuning §2.1, §3.

Method

This is a test-set study. No model is trained, and no method is proposed. The authors send prompts to GPT-4o and score the answers. Steps, in the paper's order:

  1. Language exams §2.1. Four exams are used, each scored as correct answers divided by questions:

    • USMLE Step 1 sample questions. The text reports 118 scored questions out of a 119-question sample set §2.1 tab. 1.
    • CFA Level 1 mock exam, 89 questions, zero-shot with no hints §2.1 tab. 2.
    • SAT Practice Test #1, four modules (two reading and writing, two math) of 33, 33, 27 and 27 questions tab. 3. Reading and writing and math are averaged over their two modules each tab. 4.
    • MBE sample questions, 20 questions §2.1 tab. 5.
  2. Logical reasoning §2.2. Five datasets cover three reasoning types fig. 7: EntailmentBank and bAbI task 15 (deductive); CLUTRR and bAbI task 16 (inductive); αNLI (abductive). Each dataset gives 30 items. For αNLI, 10 items come from each of the easy, medium and hard training sets. The other sets use test items §2.2. The prompts are those of López Espejel et al. (2023) §2.2. Only the first response to each question is scored §2.2.

  3. Translation §2.3. Six target languages: Spanish, Arabic, Hindi, French, Portuguese and Russian. Data come from OPUS for five languages and from the IIT Bombay English-Hindi corpus for Hindi §2.3. The study draws 500 random sentence pairs per language. Each GPT-4o output is compared with its reference translation by cosine similarity of sentence embeddings from paraphrase-MiniLM-L6-v2 §2.3:

    Pseudocode
    # Translation similarity for one language  [§2.3]
    # Input:  references (N sentences), outputs (N GPT-4o translations)
    # Output: N cosine scores; the paper's mapping to % is not stated
    
    def translation_scores(references, outputs):
        # Embed each sentence with paraphrase-MiniLM-L6-v2  [§2.3]
        reference_vectors = embed(references)   # (N, d)
        output_vectors = embed(outputs)         # (N, d)
        scores = []
        for each index i:
            # cosine(A, B) = A . B / (||A|| ||B||)  [§2.3]
            dot_product = dot(reference_vectors[i], output_vectors[i])
            lengths = norm(reference_vectors[i]) * norm(output_vectors[i])
            scores.append(dot_product / lengths)
        # How the scores become one % per language is not stated  [tab. 7]
        return scores

    The cosine formula is written as A·B / (‖A‖‖B‖) §2.3.

  4. Image tasks §3. The paper states that about 100 representative images were curated per task. Each image is sent with a text prompt that asks for an output format. Precision, recall and F1 are reported per class and as averages §3.1–§3.5. Some tasks use confusion matrices fig. 8, fig. 9, fig. 10, fig. 11, fig. 13.

    • Fruit classification, 10 classes §3.1 tab. 8 fig. 8.
    • Driver drowsiness, 2 classes, from face-region frames §3.2 tab. 9 fig. 9.
    • Crop disease, 20 classes, 100 sampled images §3.3 tab. 10 fig. 10.
    • Glaucoma, glaucoma vs normal fundus images §3.4 tab. 11 fig. 11.
    • CT scans for cancer, tumour and aneurysm §3.5 tab. 12 fig. 13.
    • Image captioning on Flickr8k, 100 images, BLEU-1 to BLEU-4 §3.6 tab. 13.
    • Few-shot test on glaucoma with 0, 1 and 4 labelled examples fig. 12.
  5. Speech §4. Emotion: 100 files sampled from the Arabic Natural Audio Dataset, three labels (happy, angry, surprised) §4.1 fig. 14. Accent: 100 files sampled from AccentDB, four accents (Bangla, Malayalam, Odia, Telugu) §4.2 fig. 15. Each audio file is sent with a prompt; the prompt wording is not stated.

  6. Multimodal §5. Visual question answering: 100 image-question pairs, with answers picked from a predefined list, tested at 0, 1, 2, 4 and 8 shots §5.1 fig. 16. Vision-language capability: MM-Vet, scored by an LLM-based evaluator that handles open-ended answers §5.2 tab. 14. Qualitative examples are in tab. 15.

Hyperparameters

Parameters that govern the evaluation. The paper sets no model hyperparameters and does not report which GPT-4o version it used.

NameValueWhereNotes
Model version or snapshotnot stated§1 §2.2Only "GPT-4o" and release timing (May 2024) are given
Access routenot stated§2.2The paper mentions "chat sessions" for reasoning runs
Temperature, top-p, max tokensnot stated—No decoding settings reported
Repeated runs, random seeds, confidence intervalsnot stated—Each item is reported once; the random sampling seed is not stated §2.3
USMLE sample size118 scored (119 in the sample set)§2.1Text says 118 and 119; see Pitfalls
CFA sample size89, zero-shot, no hints§2.1
SAT modules33, 33, 27, 27tab. 3
MBE sample size20§2.1
Reasoning items30 per dataset; αNLI 10 each from easy, medium, hard training sets§2.2
Translation items500 random pairs per language§2.3Sampling method is random; seed not stated
Translation embedderparaphrase-MiniLM-L6-v2 (sentence-transformers)§2.3Text calls the family "BERT"
Fruit / drowsiness / crop / glaucoma / CT imagesabout 100 each§3Counts in matrices differ; see Pitfalls
Caption set100 images§3.6
Few-shot glaucoma0, 1, 4 labelled examplesfig. 12
Speech sets100 emotion files, 100 accent files§4.1 §4.2
VQA100 pairs; 0, 1, 2, 4, 8 shots§5.1 fig. 16
Prompt textnot stated—Reasoning prompts follow López Espejel et al. §2.2

Results

Author-reported. Setup: exams and reasoning as in §2; image, speech and multimodal tasks as in §3–§5; no hardware or cost is reported. Baselines are GPT-3.5, GPT-4 and GPT-4V as the paper names them. Their source is given per row: most are copied from earlier reports, not re-run.

SettingMetricBaseline (source)This paperSource
USMLE Step 1 sampleaccuracyGPT-3.5 51.67%, GPT-4 90.00% (Gilson et al.; Brin et al.)83.05% (98 of 118)tab. 1
CFA Level 1 mockaccuracyGPT-3.5 58.80%, GPT-4 73.20% (Callanan et al.)85.39% (76 of 89)tab. 2
SAT reading and writingaverage accuracyGPT-3.5 83.75%, GPT-4 88.75% (OpenAI report)90.91%tab. 3 tab. 4
SAT mathaverage accuracyGPT-3.5 73.75%, GPT-4 87.50% (OpenAI report)87.04%tab. 3 tab. 4
MBE sampleaccuracyGPT-3.5 45.10%, GPT-4 75.70% (Katz et al.)75.00% (15 of 20)tab. 5
Deductive, EntailmentBankcorrect of 30GPT-3.5 25, GPT-4 27 (source not stated)29 of 30tab. 6
Deductive, bAbI task 15correct of 30GPT-3.5 26, GPT-4 3030 of 30tab. 6
Inductive, CLUTRRcorrect of 30GPT-3.5 2, GPT-4 1117 of 30tab. 6
Inductive, bAbI task 16correct of 30GPT-3.5 14, GPT-4 2830 of 30tab. 6
Abductive, αNLIcorrect of 30GPT-3.5 19, GPT-4 2527 of 30tab. 6
Translation (six languages)"translation accuracy" %noneSpanish 88, Arabic 78, Hindi 82, French 75, Portuguese 86, Russian 80tab. 7 §2.3
Fruit (10 classes)average P / R / F1none0.98 / 0.98 / 0.98§3.1 tab. 8
Drowsiness (2 classes)P / R / F1none (specialised CNNs cited without numbers)0.80 / 0.80 / 0.80§3.2 tab. 9
Crop disease (20 classes)average P / R / F1none0.77 / 0.71 / 0.68§3.3 tab. 10
Glaucoma (2 classes)average P / R / F1none0.65 / 0.62 / 0.59§3.4 tab. 11
CT cancer, tumour, aneurysmaverage P / R / F1none0.21 / 0.32 / 0.26 (cancer class 0.00)§3.5 tab. 12
Flickr8k captions (100 images)BLEU-1 / 2 / 3 / 4none0.193 / 0.095 / 0.058 / 0.031§3.6 tab. 13
Arabic emotion (3 classes)per-class P / R / F1noneNo average given; "surprised" best; "angry" 2 correct§4.1 fig. 14
AccentDB (4 accents)per-class P / R / F1noneNo average given; highest precision for Odia§4.2 fig. 15
VQA (100 pairs)accuracy by shots 0 / 1 / 2 / 4 / 8none0.35 / 0.30 / 0.35 / 0.35 / 0.36§5.1 fig. 16
MM-Vet totalscoreGPT-3.5 27.6, GPT-4 48.1, GPT-4V 67.783.9§5.2 tab. 14

MM-Vet sub-scores for GPT-4o: recognition 80.6, OCR 87.5, knowledge 75.7, language generation 77.1, spatial awareness 84.7, math 88.1 tab. 14.

Paper vs code

No official code. The paper gives no code link, and the arXiv abstract page lists none. A GitHub repository search for the title and the model name returned no results. An author-name account search returned no repository that could be tied to the authors, so none was used.

Pitfalls

  • Not a method. There is no architecture, training recipe or loss to implement. An implementer can only repeat the protocol, and the protocol is under-specified (see the rows marked "not stated" above).
  • Model identity and settings are missing. No snapshot, date, access route, temperature or token limit is stated §1 §2.2. The reasoning runs used "various chat sessions", and some answers changed between sessions, so identical prompts may not give identical results §2.2. Results on another GPT-4o version may differ.
  • Baselines were copied, not re-run. The GPT-3.5 and GPT-4 numbers come from other reports, which use other prompts and samples §2.1 tab. 2 tab. 4 tab. 5. Source for GPT-3.5 and GPT-4 in tab. 6 and tab. 14 is not stated. Do not treat these comparisons as controlled.
  • Tiny samples. One item is 1 point on a 100-item task and 3.3 points on a 30-item task. The VQA "peak" at 0.36 with eight shots is one item above 0.35 fig. 16. The text reads this as a trend §5.1; it is not evidence of one.
  • Counts that do not add up.
    • USMLE: the text gives 119 sample questions and "118 questions", with 98 correct. It states 83.1% in the text and 83.05% in the table §2.1 tab. 1.
    • Glaucoma: the text says 100 images, evenly split §3.4. The confusion matrix rows sum to 49 and 50, which is 99 fig. 11.
    • CT: the text says 100 scans §3.5. The matrix rows sum to 33 each, which is 99 fig. 13.
    • Emotion: the text says 100 files §4.1. The matrix rows sum to 32, 29 and 35, which is 96 fig. 14.
    • Fruit: the text describes an 80-image test split §3.1. The matrix rows sum to 10 per class, which is 100 fig. 8. Which images were scored is not stated.
    • None of the paper says whether items were dropped. Check before reusing these counts.
  • Impossible value in a table. Crop disease "Blossom End Rot" shows recall 1.08 tab. 10. The reported average recall of 0.71 §3.3 is consistent with averaging the table including this value, so it likely propagates.
  • Figure and table disagree on glaucoma zero-shot. Figure 12 shows the normal-class F1 near 0.03 at zero shots fig. 12. Table 11 gives 0.50 for the normal class tab. 11. The paper does not say which run each shows.
  • Accent confusion direction. The text says Malayalam recordings are often mistaken for Telugu §4.2. In the figure, Malayalam recordings predicted as Telugu number 4. In the Malayalam-predicted column, the largest off-diagonal count is 15 Telugu recordings, and 13 Bangla recordings are also predicted as Malayalam fig. 15. Check the axes before relying on the text.
  • Translation scores are not accuracy. The "translation accuracy" is a cosine similarity of sentence embeddings. The paper does not say how it became a percentage tab. 7, and it does not establish whether the references are human or machine translations §2.3. The translation direction is not stated. No human check and no standard MT metric is reported.
  • Reasoning items. Only the first answer counts §2.2. The αNLI items come from training splits §2.2, so the αNLI numbers do not come from the test split. Answer parsing and failure handling are not stated.
  • Image and speech prompts and parsing. The prompt wording and the rule for reading the model's answer are not stated for any image or audio task §3.2 §4.1. The drowsiness task asks for JSON output §3.2. Malformed replies are not discussed.
  • Ground truth for speech. The emotion labels come from 18 listeners per recording. How the perceived label was chosen is not stated §4.1. The accent labels come from the dataset; how they were assigned is not stated §4.2.
  • Captioning metric. The BLEU variant, smoothing, tokenisation and number of reference captions per image are not stated §3.6.
  • VQA and MM-Vet scoring. The predefined VQA answer list and its matching rule are not stated §5.1. The LLM that judges MM-Vet answers is not named §5.2. Both affect the reported numbers.
  • Few-shot claim is thin. The abstract's claim of excelling at few-shot tasks rests on the glaucoma plot and the VQA bars fig. 12 fig. 16. The glaucoma normal class goes from near zero to one example and then stays flat fig. 12. The VQA change is one item.
  • No uncertainty. No repeated runs, variance or confidence intervals are reported anywhere in the paper.

Lineage

  • Builds on: the MM-Vet benchmark (Yu et al., 2023; [arXiv:2308.02490]) for the vision-language scores §5.2; the reasoning prompts of López Espejel et al. (2023), a journal article without an arXiv id §2.2.
  • Compared against: GPT-3.5 and GPT-4 numbers taken from prior reports across the exam and reasoning tables tab. 1–tab. 6; GPT-4V in tab. 14.
  • Superseded by: not stated in this paper.

Source and license

Adapted from “Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency” by Sakib Shahriar, Brady Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, Laiba Batool (arXiv:2407.09519), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.