MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

In plain words

Checking whether an AI model did a generation job well is hard when the output is a picture, a sound clip, or a mix of text and pictures. Automatic scorers often disagree with people, and asking a model to grade another model is unreliable unless the question is very precise.

MMMG is a test suite built around that problem. Each test asks for something specific, such as an object count, a text label on an object, a given instrument in a song, or a spoken sentence with a required voice. Each test comes with a way to check the result: a small program, a vision or audio model that answers a narrow multiple-choice or yes/no question, or a comparison with a reference sample. The authors then checked these automatic checks against human graders to see whether they agree.

Results are reported for 29 models. Image generators do best on single pictures, while models that mix text and images in one answer do much worse, and sound and music generation has the most room left to improve.

TL;DR

MMMG is a benchmark with 55 tasks and 1288 instructions over four modality combinations, where each task has a verifiable or near-verifiable check. The authors report 94.4% average agreement between their automatic evaluation and human annotators. Their results cover 29 models; the best image model is GPT Image at 70.7%, and interleaved and audio generation score much lower. Use it when you need a reliability-audited multimodal generation benchmark, and read the pitfalls below before comparing scores with other papers.

Problem

Automatic evaluation of multimodal generation often misaligns with human judgment, and multimodal-language-model judges are used without validation §1. Most existing benchmarks cover a single modality §1. The paper's premise is that text generation has verifiable instructions that programs can check, whereas images and audio rarely reduce to a simple rule §1. The authors propose tasks that are either verifiable by program or near-verifiable by strengthening the task conditions (reference inputs, explicit rubrics) so that a judge's job reduces to a simple, objective check §1 §3.1.

Method

The benchmark is built in three stages §3.1:

1. Task creation. An initial pool of 78 candidate task templates is narrowed to 60 by a feasibility check that requires at least one reliable evaluation method §3.1.
2. Instruction synthesis and validation. Experts write a template and five seed instructions per task. GPT-4o is prompted to produce 10 candidate instructions per task. Candidates are filtered for ambiguity, unrealism and redundancy, and then checked for verifiability by comparing automatic judgments with human judgments on sampled outputs. The loop repeats until about 40 instructions per task are accepted; the paper reports that about 10% of generated instructions pass §3.1.
3. Postprocessing. Two annotators judge task realism and six tasks are removed §3.1. The remaining instructions are paraphrased and each paraphrase is manually checked for semantic equivalence §3.1.

The end result is 1288 instructions across 55 tasks and four modality combinations §3.1 tab. 7. Table 1 lists every task with its subtask, input and output modalities, instruction count and evaluation method tab. 1.

Evaluation methods §3.2. Each task uses the method that best matched human judgment in the paper's study tab. 1:

• VLM judge (image tasks). A vision-language model answers a question per criterion. Two kinds of rubric are used. Format rubrics constrain the answer: chain-of-thought before any yes/no, and multiple-choice for counting and spatial questions §3.2 app. C.3. Content rubrics list required objects, attributes and relations, plus explicit negative conditions in concrete wording §3.2. The paper's point is that human-written rubrics reduce hallucinated and near-miss acceptances relative to automatically generated VQA pairs such as TIFA §3.2.
• Image similarity. DreamSim for perceptual similarity; SSIM where exact matching is needed §3.2. Image editing combines two checks: DreamSim or VQA on the edited region and SSIM on the unchanged area §3.2 fig. 1.
• Audio similarity. Average top-10 CLAP cosine similarity against reference clips from ESC-50, OpenMIC-2018 and GTZAN §3.2 fig. 1.
• Audio models. WavLM for speaker similarity, Whisper for transcription, a fine-tuned Wav2Vec checkpoint for gender, and BeatThis for beat tracking and tempo §3.2 fig. 1.
• Programs. PIL for image checks; Librosa and Praat for pitch, intensity and speed; the IFEval implementation for textual constraints; word accuracy for exact text rendering and text-to-speech §3.2 app. C.3.

Scoring §3.2. Binary outcomes are mapped to 0 or 1 and task scores are macro-averaged, following GenEval §3.2. For CLAP scores, the paper selects a task-specific threshold so that the score's sensitivity matches the binary VLM judgments §3.2 app. C.1.

Per-task procedures app. C.3:

• Solid colour fill (program). Compare the crop's mean RGB to a reference (relative deviation above 15% scores zero), compute SSIM against a generated solid patch, then subtract a penalty for pixels in the margin that match the colour app. C.3.
• Image editing (program + VLM/SSIM). Crop the labelled edit area and compare it with a reference or a VLM judgement; compare the rest of the image with SSIM; the score is the product of the two app. C.3.
• Sound generation. Crop the first or last 4 seconds for begin/end tasks; split by librosa.effects.split for silence tasks and score each segment with CLAP app. C.3.
• Music tempo and intensity. Tempo from BeatThis beats converted to BPM; intensity from the slope and fit of the RMS energy in the first and last 4 seconds, with a clear upward or downward trend required app. C.3.
• Speech. Pitch from Parselmouth; speed in words per minute (English) or characters per minute (Chinese) from Whisper; textual constraints from Whisper-normalised transcripts scored with IFEval tools; BLEU for speech translation and word accuracy for other speech tasks app. C.3.

Generation protocol app. C.2. Sample four outputs per instruction; the paper follows GenEval for this app. C.2. Temperature is 0 for multimodal language models and 200 sampling steps for diffusion models; other settings are left at defaults app. C.2. Agent-based models receive a system prompt that asks for placeholder function calls; placeholders are resolved in dependency order by topological sort, and cycles are reported as errors app. C.2 tab. 4 tab. 5.

# Evaluate one task  [§3.2] [app. C.2]
# Input:  instructions (list), model (generator), judge (VLM, audio or program)
# Output: task score in [0, 1]

def evaluate_task(instructions, model, judge):
    # Generate 4 samples per instruction  [app. C.2]
    outputs = [model.generate(inst, n=4) for inst in instructions]
    scores = []
    for inst, samples in zip(instructions, outputs):
        # Score each sample with the task's check  [§3.2]
        for sample in samples:
            scores.append(judge.check(inst, sample))  # 0 or 1, or similarity
    # Binary results are macro-averaged  [§3.2]
    return mean(scores)

Hyperparameters

Results

Author-reported. Setup: 29 generation models evaluated with the benchmark's own judges, with humans annotating 978 instructions and 2596 questions §4.1.

Human alignment §4.1 tab. 8:

Benchmarking (headline) §1 §4.2 fig. 2:

The paper also reports that the agent Gemini 2.5 + GPT Image improves over GPT Image alone by 21.1% on interleaved image-text §4.2, and that Gemini 2.5 planning gives a 51.8% improvement over GPT-4o planning with GPT Image app. D.2. Speech-text generation reports a Spirit LM result of essentially zero on most tasks app. D.2.

Paper vs code

Code at commit f7de1cf7b954c9a7a7357efa7fa21d713262a839 of yaojh18/MMMG (default branch master). The repository has no LICENSE file, so no code is quoted here and the code is cited by path only. The owner name appears to match the first author's name (an inference, not stated in the paper); the link appears only as a commented-out line in the LaTeX source of the abstract. The compiled paper links an anonymous mirror instead §A.3.

• Retry count for image models. Paper: 4 retries for multimodal language models app. C.2. Code: 2 retries in the image model wrappers model_image.py model_image.py; 4 only in the interleaved model model_interleaved.py. Retry counts are not the same across models.
• Temperature for the interleaved agent. Paper: temperature 0 app. C.2. Code: the agent call uses temperature 0.1 model_interleaved.py, and one branch samples with temperature 0.5 and do_sample=True model_interleaved.py model_interleaved.py.
• Judge thresholds. Paper: task-specific thresholds chosen to align with human judgment, values not stated §3.2. Code: fixed defaults per class, 0.68 eval_audio.py, 0.93 eval_audio.py eval_interleaved.py, and others at 0.62 and 0.69 eval_audio.py eval_audio.py. The optimal-threshold search is commented out eval_interleaved.py.
• Interleaved accuracy formula. Paper: macro-averaged binary scores §3.2. Code: per instruction, the auto-score mean is multiplied by a product of threshold tests across generations eval_interleaved.py. The paper does not state this combination.
• Repository counts. The README describes 49 tasks, 937 instructions and 94.3% agreement readme.md. The paper reports 55 tasks, 1288 instructions and 94.4% §1. The code therefore describes a different version from the paper.
• Sample size. Paper: 4 samples per instruction app. C.2. Code default matches eval_pipeline.py.

Pitfalls

• Task count is inconsistent. Text says 78 candidate templates, then "the complete list of 76 tasks", then 60 after feasibility and six removed (which gives 54) §3.1. The abstract, the statistics table and the conclusion say 55 tab. 7. Use 55 with the table, but do not infer the list from the text.
• Model count differs across tables. The image table caption says 10 models tab. 9, and the separability analysis says 10 app. D.3. The confidence-interval caption says 9 tab. 13, and the leaderboard analysis uses nine models §4.3 tab. 3. Check which set a number comes from.
• Average agreement differs. The abstract gives 94.4%, and the per-modality figures in the introduction average to about 93.4%. The human table caption gives 0.944, while its average row shows 0.935 for GPT-4o and 0.980 IAA; the text gives IAA 97.1% §4.1 tab. 8.
• Judges are validated on 41 tasks only. Human validation covers 41 tasks and 978 instructions §1; the remaining tasks are program-checked or lack a matching human study. Do not generalise the 94.4% to tasks outside that set.
• Judge thresholds are not reported as numbers in the paper, but the code uses fixed values that differ by class (see Paper vs code). Reproducing the reported accuracies needs the code's values, or a re-tuned threshold per task.
• Model judges depend on proprietary APIs. GPT-4o, Gemini 2.5 and the Chinese Whisper fine-tune set the evaluation; the paper says open models lag app. A.1. A re-run on open models changes the scores.
• CLAP reference sets are small. Audio similarity compares against clips from ESC-50, OpenMIC-2018 and GTZAN; the paper's error analysis says CLAP struggles with synthetic audio and that a larger reference set is needed app. D.4 tab. 20.
• Confidence intervals are relative and averaged. Reported CI figures are averages across tasks and models, and the maximum is 10.12% across tasks app. D.3. Per-task variance is larger for some models.
• Leaderboard correlation is small-sample. Correlation uses nine image models and 400 of 1600 GenAI-Bench instructions, sampled because of budget §4.3. Do not read the Spearman value of 0.561 as a population estimate.
• Costs. Evaluation needs one A10-class GPU for open models and paid APIs for proprietary ones; the average API cost quoted is about $1.1 per task at sample size 4, and generation for one model can take about 3 hours per task app. B.4.
• Licence of the code is unknown. The repository has no LICENSE. Do not copy code from it, and treat any reuse as needing a licence decision.

Lineage

• Builds on: the verifiable-instruction paradigm of IFEval §1; the per-task evaluation design of GenEval for image tasks §3.2 app. B; the interleaved-generation benchmark ISG-Bench for 3D transformation and sourcing §1 app. B.1.
• Compared against: GenEval, DrawBench and GenAI-Bench on the image Chatbot Arena correlation tab. 3; MMIE, CoMM and ISG-Bench for interleaved generation §1 tab. 2.
• Superseded by: not stated.