Paper · 2024-06-19 · cs.AI, cs.CL
Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
Author-run evaluation of GPT-4o on exams, logical reasoning, translation, image classification, captioning, VQA, speech emotion and accent tasks, and MM-Vet, using 20 to 500 items per task. Strongest on exams and deductive reasoning (83.05% on a USMLE Step 1 sample, 85.39% on a CFA Level 1 mock); weakest on CT cancer classes and accent recognition. Author-reported.
LLM evaluationMultimodal language modelBenchmark validityFew-shot learningevaluationbenchmarkmultimodalvisionspeechreasoningtranslation
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
This paper does not propose a new model. It tests GPT-4o, a model from OpenAI that handles text, images and audio, on a broad set of everyday benchmark-style tasks: exam questions, logic puzzles, translation, image recognition, sound and speech classification, and questions about pictures. Each task gets a plain prompt, and the answers are compared with a known correct answer.
The results are mixed. The model does well on exams and on text-based reasoning. It is weak on several image and audio tasks, such as the CT brain scan categories and accent recognition. The samples are small, each task was run once, and some comparison numbers come from earlier papers rather than from the authors' own runs. So the numbers are a snapshot of one model at one time, not a settled ranking.
Treat the card as a list of checks to run yourself before trusting this model in your own product, not as a verdict on the model.
TL;DR
- An evaluation study, not a method: GPT-4o is tested across language, vision, speech and multimodal tasks, with about 20 to 500 items per task §2–§5.
- Strong results on exams (83.05% on a USMLE Step 1 sample tab. 1) and on deductive reasoning tab. 6; weak results on CT cancer classes tab. 12 and accent recognition fig. 15.
- Single runs, small samples, no code, and some baselines copied from other papers. Use the numbers as a starting point, not as targets to match.
Problem
The authors state that GPT-4o's language, vision, speech and multimodal abilities had not been formally evaluated in the literature, and that earlier GPT-4 and GPT-4V evaluations were limited in scope §1, §1.2. They ask for broader benchmarks, error analysis and human judgement §6.2. The study assumes public sample datasets and zero-shot or few-shot prompting, with no training or fine-tuning §2.1, §3.
Method
This is a test-set study. No model is trained, and no method is proposed. The authors send prompts to GPT-4o and score the answers. Steps, in the paper's order:
-
Language exams §2.1. Four exams are used, each scored as correct answers divided by questions:
- USMLE Step 1 sample questions. The text reports 118 scored questions out of a 119-question sample set §2.1 tab. 1.
- CFA Level 1 mock exam, 89 questions, zero-shot with no hints §2.1 tab. 2.
- SAT Practice Test #1, four modules (two reading and writing, two math) of 33, 33, 27 and 27 questions tab. 3. Reading and writing and math are averaged over their two modules each tab. 4.
- MBE sample questions, 20 questions §2.1 tab. 5.
-
Logical reasoning §2.2. Five datasets cover three reasoning types fig. 7: EntailmentBank and bAbI task 15 (deductive); CLUTRR and bAbI task 16 (inductive); αNLI (abductive). Each dataset gives 30 items. For αNLI, 10 items come from each of the easy, medium and hard training sets. The other sets use test items §2.2. The prompts are those of López Espejel et al. (2023) §2.2. Only the first response to each question is scored §2.2.
-
Translation §2.3. Six target languages: Spanish, Arabic, Hindi, French, Portuguese and Russian. Data come from OPUS for five languages and from the IIT Bombay English-Hindi corpus for Hindi §2.3. The study draws 500 random sentence pairs per language. Each GPT-4o output is compared with its reference translation by cosine similarity of sentence embeddings from paraphrase-MiniLM-L6-v2 §2.3:
Pseudocode# Translation similarity for one language [§2.3] # Input: references (N sentences), outputs (N GPT-4o translations) # Output: N cosine scores; the paper's mapping to % is not stated def translation_scores(references, outputs): # Embed each sentence with paraphrase-MiniLM-L6-v2 [§2.3] reference_vectors = embed(references) # (N, d) output_vectors = embed(outputs) # (N, d) scores = [] for each index i: # cosine(A, B) = A . B / (||A|| ||B||) [§2.3] dot_product = dot(reference_vectors[i], output_vectors[i]) lengths = norm(reference_vectors[i]) * norm(output_vectors[i]) scores.append(dot_product / lengths) # How the scores become one % per language is not stated [tab. 7] return scoresThe cosine formula is written as A·B / (‖A‖‖B‖) §2.3.
-
Image tasks §3. The paper states that about 100 representative images were curated per task. Each image is sent with a text prompt that asks for an output format. Precision, recall and F1 are reported per class and as averages §3.1–§3.5. Some tasks use confusion matrices fig. 8, fig. 9, fig. 10, fig. 11, fig. 13.
- Fruit classification, 10 classes §3.1 tab. 8 fig. 8.
- Driver drowsiness, 2 classes, from face-region frames §3.2 tab. 9 fig. 9.
- Crop disease, 20 classes, 100 sampled images §3.3 tab. 10 fig. 10.
- Glaucoma, glaucoma vs normal fundus images §3.4 tab. 11 fig. 11.
- CT scans for cancer, tumour and aneurysm §3.5 tab. 12 fig. 13.
- Image captioning on Flickr8k, 100 images, BLEU-1 to BLEU-4 §3.6 tab. 13.
- Few-shot test on glaucoma with 0, 1 and 4 labelled examples fig. 12.
-
Speech §4. Emotion: 100 files sampled from the Arabic Natural Audio Dataset, three labels (happy, angry, surprised) §4.1 fig. 14. Accent: 100 files sampled from AccentDB, four accents (Bangla, Malayalam, Odia, Telugu) §4.2 fig. 15. Each audio file is sent with a prompt; the prompt wording is not stated.
-
Multimodal §5. Visual question answering: 100 image-question pairs, with answers picked from a predefined list, tested at 0, 1, 2, 4 and 8 shots §5.1 fig. 16. Vision-language capability: MM-Vet, scored by an LLM-based evaluator that handles open-ended answers §5.2 tab. 14. Qualitative examples are in tab. 15.
Hyperparameters
Parameters that govern the evaluation. The paper sets no model hyperparameters and does not report which GPT-4o version it used.
| Name | Value | Where | Notes |
|---|---|---|---|
| Model version or snapshot | not stated | §1 §2.2 | Only "GPT-4o" and release timing (May 2024) are given |
| Access route | not stated | §2.2 | The paper mentions "chat sessions" for reasoning runs |
| Temperature, top-p, max tokens | not stated | — | No decoding settings reported |
| Repeated runs, random seeds, confidence intervals | not stated | — | Each item is reported once; the random sampling seed is not stated §2.3 |
| USMLE sample size | 118 scored (119 in the sample set) | §2.1 | Text says 118 and 119; see Pitfalls |
| CFA sample size | 89, zero-shot, no hints | §2.1 | |
| SAT modules | 33, 33, 27, 27 | tab. 3 | |
| MBE sample size | 20 | §2.1 | |
| Reasoning items | 30 per dataset; αNLI 10 each from easy, medium, hard training sets | §2.2 | |
| Translation items | 500 random pairs per language | §2.3 | Sampling method is random; seed not stated |
| Translation embedder | paraphrase-MiniLM-L6-v2 (sentence-transformers) | §2.3 | Text calls the family "BERT" |
| Fruit / drowsiness / crop / glaucoma / CT images | about 100 each | §3 | Counts in matrices differ; see Pitfalls |
| Caption set | 100 images | §3.6 | |
| Few-shot glaucoma | 0, 1, 4 labelled examples | fig. 12 | |
| Speech sets | 100 emotion files, 100 accent files | §4.1 §4.2 | |
| VQA | 100 pairs; 0, 1, 2, 4, 8 shots | §5.1 fig. 16 | |
| Prompt text | not stated | — | Reasoning prompts follow López Espejel et al. §2.2 |
Results
Author-reported. Setup: exams and reasoning as in §2; image, speech and multimodal tasks as in §3–§5; no hardware or cost is reported. Baselines are GPT-3.5, GPT-4 and GPT-4V as the paper names them. Their source is given per row: most are copied from earlier reports, not re-run.
| Setting | Metric | Baseline (source) | This paper | Source |
|---|---|---|---|---|
| USMLE Step 1 sample | accuracy | GPT-3.5 51.67%, GPT-4 90.00% (Gilson et al.; Brin et al.) | 83.05% (98 of 118) | tab. 1 |
| CFA Level 1 mock | accuracy | GPT-3.5 58.80%, GPT-4 73.20% (Callanan et al.) | 85.39% (76 of 89) | tab. 2 |
| SAT reading and writing | average accuracy | GPT-3.5 83.75%, GPT-4 88.75% (OpenAI report) | 90.91% | tab. 3 tab. 4 |
| SAT math | average accuracy | GPT-3.5 73.75%, GPT-4 87.50% (OpenAI report) | 87.04% | tab. 3 tab. 4 |
| MBE sample | accuracy | GPT-3.5 45.10%, GPT-4 75.70% (Katz et al.) | 75.00% (15 of 20) | tab. 5 |
| Deductive, EntailmentBank | correct of 30 | GPT-3.5 25, GPT-4 27 (source not stated) | 29 of 30 | tab. 6 |
| Deductive, bAbI task 15 | correct of 30 | GPT-3.5 26, GPT-4 30 | 30 of 30 | tab. 6 |
| Inductive, CLUTRR | correct of 30 | GPT-3.5 2, GPT-4 11 | 17 of 30 | tab. 6 |
| Inductive, bAbI task 16 | correct of 30 | GPT-3.5 14, GPT-4 28 | 30 of 30 | tab. 6 |
| Abductive, αNLI | correct of 30 | GPT-3.5 19, GPT-4 25 | 27 of 30 | tab. 6 |
| Translation (six languages) | "translation accuracy" % | none | Spanish 88, Arabic 78, Hindi 82, French 75, Portuguese 86, Russian 80 | tab. 7 §2.3 |
| Fruit (10 classes) | average P / R / F1 | none | 0.98 / 0.98 / 0.98 | §3.1 tab. 8 |
| Drowsiness (2 classes) | P / R / F1 | none (specialised CNNs cited without numbers) | 0.80 / 0.80 / 0.80 | §3.2 tab. 9 |
| Crop disease (20 classes) | average P / R / F1 | none | 0.77 / 0.71 / 0.68 | §3.3 tab. 10 |
| Glaucoma (2 classes) | average P / R / F1 | none | 0.65 / 0.62 / 0.59 | §3.4 tab. 11 |
| CT cancer, tumour, aneurysm | average P / R / F1 | none | 0.21 / 0.32 / 0.26 (cancer class 0.00) | §3.5 tab. 12 |
| Flickr8k captions (100 images) | BLEU-1 / 2 / 3 / 4 | none | 0.193 / 0.095 / 0.058 / 0.031 | §3.6 tab. 13 |
| Arabic emotion (3 classes) | per-class P / R / F1 | none | No average given; "surprised" best; "angry" 2 correct | §4.1 fig. 14 |
| AccentDB (4 accents) | per-class P / R / F1 | none | No average given; highest precision for Odia | §4.2 fig. 15 |
| VQA (100 pairs) | accuracy by shots 0 / 1 / 2 / 4 / 8 | none | 0.35 / 0.30 / 0.35 / 0.35 / 0.36 | §5.1 fig. 16 |
| MM-Vet total | score | GPT-3.5 27.6, GPT-4 48.1, GPT-4V 67.7 | 83.9 | §5.2 tab. 14 |
MM-Vet sub-scores for GPT-4o: recognition 80.6, OCR 87.5, knowledge 75.7, language generation 77.1, spatial awareness 84.7, math 88.1 tab. 14.
Paper vs code
No official code. The paper gives no code link, and the arXiv abstract page lists none. A GitHub repository search for the title and the model name returned no results. An author-name account search returned no repository that could be tied to the authors, so none was used.
Pitfalls
- Not a method. There is no architecture, training recipe or loss to implement. An implementer can only repeat the protocol, and the protocol is under-specified (see the rows marked "not stated" above).
- Model identity and settings are missing. No snapshot, date, access route, temperature or token limit is stated §1 §2.2. The reasoning runs used "various chat sessions", and some answers changed between sessions, so identical prompts may not give identical results §2.2. Results on another GPT-4o version may differ.
- Baselines were copied, not re-run. The GPT-3.5 and GPT-4 numbers come from other reports, which use other prompts and samples §2.1 tab. 2 tab. 4 tab. 5. Source for GPT-3.5 and GPT-4 in tab. 6 and tab. 14 is not stated. Do not treat these comparisons as controlled.
- Tiny samples. One item is 1 point on a 100-item task and 3.3 points on a 30-item task. The VQA "peak" at 0.36 with eight shots is one item above 0.35 fig. 16. The text reads this as a trend §5.1; it is not evidence of one.
- Counts that do not add up.
- USMLE: the text gives 119 sample questions and "118 questions", with 98 correct. It states 83.1% in the text and 83.05% in the table §2.1 tab. 1.
- Glaucoma: the text says 100 images, evenly split §3.4. The confusion matrix rows sum to 49 and 50, which is 99 fig. 11.
- CT: the text says 100 scans §3.5. The matrix rows sum to 33 each, which is 99 fig. 13.
- Emotion: the text says 100 files §4.1. The matrix rows sum to 32, 29 and 35, which is 96 fig. 14.
- Fruit: the text describes an 80-image test split §3.1. The matrix rows sum to 10 per class, which is 100 fig. 8. Which images were scored is not stated.
- None of the paper says whether items were dropped. Check before reusing these counts.
- Impossible value in a table. Crop disease "Blossom End Rot" shows recall 1.08 tab. 10. The reported average recall of 0.71 §3.3 is consistent with averaging the table including this value, so it likely propagates.
- Figure and table disagree on glaucoma zero-shot. Figure 12 shows the normal-class F1 near 0.03 at zero shots fig. 12. Table 11 gives 0.50 for the normal class tab. 11. The paper does not say which run each shows.
- Accent confusion direction. The text says Malayalam recordings are often mistaken for Telugu §4.2. In the figure, Malayalam recordings predicted as Telugu number 4. In the Malayalam-predicted column, the largest off-diagonal count is 15 Telugu recordings, and 13 Bangla recordings are also predicted as Malayalam fig. 15. Check the axes before relying on the text.
- Translation scores are not accuracy. The "translation accuracy" is a cosine similarity of sentence embeddings. The paper does not say how it became a percentage tab. 7, and it does not establish whether the references are human or machine translations §2.3. The translation direction is not stated. No human check and no standard MT metric is reported.
- Reasoning items. Only the first answer counts §2.2. The αNLI items come from training splits §2.2, so the αNLI numbers do not come from the test split. Answer parsing and failure handling are not stated.
- Image and speech prompts and parsing. The prompt wording and the rule for reading the model's answer are not stated for any image or audio task §3.2 §4.1. The drowsiness task asks for JSON output §3.2. Malformed replies are not discussed.
- Ground truth for speech. The emotion labels come from 18 listeners per recording. How the perceived label was chosen is not stated §4.1. The accent labels come from the dataset; how they were assigned is not stated §4.2.
- Captioning metric. The BLEU variant, smoothing, tokenisation and number of reference captions per image are not stated §3.6.
- VQA and MM-Vet scoring. The predefined VQA answer list and its matching rule are not stated §5.1. The LLM that judges MM-Vet answers is not named §5.2. Both affect the reported numbers.
- Few-shot claim is thin. The abstract's claim of excelling at few-shot tasks rests on the glaucoma plot and the VQA bars fig. 12 fig. 16. The glaucoma normal class goes from near zero to one example and then stays flat fig. 12. The VQA change is one item.
- No uncertainty. No repeated runs, variance or confidence intervals are reported anywhere in the paper.
Lineage
- Builds on: the MM-Vet benchmark (Yu et al., 2023; [arXiv:2308.02490]) for the vision-language scores §5.2; the reasoning prompts of López Espejel et al. (2023), a journal article without an arXiv id §2.2.
- Compared against: GPT-3.5 and GPT-4 numbers taken from prior reports across the exam and reasoning tables tab. 1–tab. 6; GPT-4V in tab. 14.
- Superseded by: not stated in this paper.
Source and license
Adapted from “Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency” by Sakib Shahriar, Brady Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, Laiba Batool (arXiv:2407.09519), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.