wikillm

Paper · 2024-03-12 · cs.CV

In-context learning enables multimodal large language models to classify cancer pathology images

Shows GPT-4V can classify histopathology tiles from a few example images in the prompt. Examples are picked by nearest neighbours in Phikon embedding space. Ten-shot kNN reaches 83.3% on MHIST and 88.3% on PatchCamelyon, above four ImageNet-initialised classifiers trained on the same images on those two tasks. Author-reported.

In-context learningFew-shot example selectionMultimodal language modelFew-shot learningpathologymedical-imagingin-context-learningfew-shotvision-language-modelnearest-neighbourgpt-4vevaluation

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Training a computer to recognise tissue under a microscope usually means collecting many labelled images and training a model on them, or adjusting a large pretrained model. This paper asks whether a general-purpose model that reads both pictures and text can learn a task without any training. The authors show it a handful of labelled example images inside the prompt, then ask it to label a new image, in the same way a person might be shown a few examples before a quiz.

The examples are not chosen at random. For each candidate label, the method picks the example images that look most similar to the new one, using an image-feature model that was trained on pathology slides. Ask a model with more examples and it does better, and with the right examples it is competitive with small image classifiers that were trained on the same images.

The tests cover three tasks: telling tissue types apart in colorectal cancer slides, telling two kinds of colon polyp apart, and spotting breast cancer in lymph-node slides. The model also gives a written reason for each answer, which a pathologist can read. The results come from the authors' own experiments, on small test sets, so they show what the method can do rather than a settled benchmark.

TL;DR

GPT-4V, a vision-language model, classifies three histopathology tasks from few-shot image examples in the prompt, with no weight updates. Examples chosen by nearest-neighbour search in a pathology-embedding space beat random ones. With ten examples per class, GPT-4V beats four ImageNet classifiers, each retrained on the same images for every test image, on MHIST and PatchCamelyon. On the eight-label CRC100K task the retrained classifiers are ahead. The experiment is API-based, used test sets of 60 to 120 images, and ran in late 2023 on a preview model.

Problem

Medical image classifiers need labelled, task-specific data, either to train from scratch or to fine-tune a foundation model, and both steps are costly § Introduction. Few-shot prompting, which shows a language model worked examples in the prompt, remains underexplored for medical images § Abstract. Text-only in-context learning has been studied widely, but the authors note that for biomedical images the question was open § Results (ICL improves accuracy). Annotated data is scarce for rare tumour types, which is why a method that needs only a few examples is of interest § Introduction.

Method

Setting and conventions. The authors evaluate three histopathology benchmarks: CRC100K tissue classes (tumour versus normal mucosa in the binary setting, eight labels in the multiclass setting), MHIST (hyperplastic polyp versus sessile serrated adenoma) and PatchCamelyon (metastatic breast cancer in lymph-node sections) § Methods (Datasets) fig. 1. Section numbers are absent from the compiled paper, so sections are cited by title, for example [§ Methods] (Datasets).

Test sets. Binary experiments use 60 images per task, balanced 1:1 across labels. The multiclass CRC100K experiment uses 120 images § Methods (Datasets). For MHIST, the test images are restricted to cases where all seven pathologists agreed § Methods (Datasets).

Model. GPT-4V is called through the chat-completions endpoint of the OpenAI Python API, model name gpt-4-vision-preview, between 15 November and 3 December 2023. Temperature is 0.1; no other model settings are changed § Methods (GPT-4V Model Specifications).

Prompt structure. Each request has a system message and a user message. The system message sets the role as a pathology assistant and asks for output as a JSON object with three fields: thoughts (free-text reasoning), answer (one label from a fixed list) and score (a confidence between 0 and 1). The prompt also lists examples of poor answers to avoid § Methods (Prompting) app. B.

Zero-shot prompt. The user message lists the possible labels, asks for step-by-step reasoning in the style of chain-of-thought prompting, and adds dataset-specific rules. For CRC100K these rules say to prefer debris over cancer when cells are dead or the tissue is disrupted, to prefer stroma over muscle when collagen is visible, and to prefer lymphocytes when they are a substantial share of the cells § Methods (Prompting) app. B. The authors chose these rules after seeing the model over-call tumour on CRC100K § Methods (Prompting).

Few-shot prompt. Example images are added before the target image. Each example is preceded by a short text that names its label, starting with "The following image contains" § Methods (Prompting). Shots are interleaved across labels: one example of each label, then the next round, repeated k times. The authors use k in {1, 3, 5, 10}. Each example is ordered so that the most similar image is shown first § Methods (kNN-based few-shot image sampling). The few-shot prompt text is divided into a part before and a part after the separator line, and the target image sits between them vision.py.

Choosing the shots. Two strategies are compared. Random sampling draws examples at random for each label. kNN sampling is Algorithm 1 alg. 1 (Appendix A): each pool image is embedded once with the Phikon ViT-B pathology encoder (768 values per image), and for each label the k pool images with the highest cosine similarity to the target are selected. The target itself is excluded alg. 1. The authors also state that tiles from the same patient are removed when patient identity is known § Methods (kNN-based few-shot image sampling). The code does not do this; see Paper vs code.

Pseudocode
# k-NN few-shot sampling per label  [alg. 1]
# Input:  pool (id, label, embedding of 768 values); target (id, embedding)
# Output: dict label -> k pool ids, most similar first

def sample_shots(target, pool, labels, k):
    shots = {}
    for label in labels:
        # Candidates: pool images with this label, never the target itself
        # [alg. 1] [code: knn_dataset.py#L75]
        candidates = [
            image for image in pool
            if image.label == label and image.id != target.id
        ]
        # Cosine similarity of each candidate to the target embedding
        # [alg. 1] [code: knn_dataset.py#L87-L88]
        scores = [
            cosine(target.embedding, image.embedding)
            for image in candidates
        ]
        # Keep the k highest scores; order is most similar first, or last
        # when most_similar_last is set  [code: knn_dataset.py#L95-L98]
        shots[label] = top_k(candidates, scores, k)
    return shots
Pseudocode
# Few-shot message list  [§ Methods] (Prompting)
# Input:  system prompt, user prompt before and after the separator,
#         shots per label, target image, k
# Output: list of chat messages

def build_messages(system, user_pre, user_post, shots, target, k):
    # System message sets the role and the JSON answer format [app. B]
    messages = [system_message(system)]
    # Prompt text before the separator goes first  [code: vision.py#L124]
    messages.append(user_message(user_pre))
    # One example per label in each round, for k rounds
    # [code: vision.py#L156-L163]
    for round_index in range(k):
        for label, examples in shots.items():
            if round_index < len(examples):
                messages.append(label_example(label, examples[round_index]))
    # Target image and the rest of the question  [code: vision.py#L125]
    messages.append(user_message(user_post, target))
    return messages

Comparison models. Four image classifiers are trained, one new model per target image: ResNet-18, ResNet-50, a Tiny ViT and a Small ViT. Each starts from ImageNet weights and is trained on the same images that the GPT-4V prompt received, with a fresh classification layer § Methods (Tile-Level Classification Benchmarks). Training runs for ten epochs with Adam at learning rate 0.001 and cross-entropy loss. Accuracy is unweighted § Methods (Tile-Level Classification Benchmarks). The authors count 3,600 trained models in total § Methods (Tile-Level Classification Benchmarks).

Pseudocode
# Per-target baseline classifier  [§ Methods] (Tile-Level Classification
# Benchmarks)
# Input:  backbone name, shot images with labels, number of classes
# Output: classifier that predicts the target's class

def train_baseline(backbone, shot_images, shot_labels, num_classes):
    # Start from ImageNet weights, with a head of num_classes outputs
    # [§ Methods] (Tile-Level Classification Benchmarks)
    # [code: VisionModels/train_classifier.ipynb#L56]
    model = create_pretrained(backbone, num_classes)
    # Adam at learning rate 0.001, cross-entropy
    # [code: VisionModels/train_classifier.ipynb#L40-L41]
    # [code: VisionModels/train_classifier.ipynb#L77]
    optimizer = adam(model.parameters(), learning_rate=0.001)
    # Ten passes over the shot images  [§ Methods] (Tile-Level Classification
    # Benchmarks)
    for epoch in range(10):
        for batch_images, batch_labels in batches(shot_images, shot_labels):
            loss = cross_entropy(model(batch_images), batch_labels)
            optimizer.step(loss)
    return model

Confidence intervals. Intervals come from 100,000 bootstrap resamples of the test set, taking the 2.5% and 97.5% points fig. 2 fig. 3 (captions). They reflect test-set sampling only, not variation across training seeds.

Compute. All experiments were run on an Apple MacBook Pro M2 Max with 96 GB of memory § Methods (Tile-Level Classification Benchmarks).

Hyperparameters

NameValueWhereNotes
Modelgpt-4-vision-preview (GPT-4V), chat completions§ Methods (GPT-4V Model Specifications)Preview endpoint, used 15 Nov to 3 Dec 2023
Temperature0.1§ Methods (GPT-4V Model Specifications) vision.pySet in the code; the config files say 1 but are not passed to the call (see Paper vs code)
Random seed (API)42vision.pyNot stated in the paper
Max output tokens4096vision.pyNot stated in the paper
Top-p, frequency penaltynot statednot statedNot set in the code call
Output formatJSON object with thoughts, answer, score§ Methods (Prompting)JSON mode is not enforced by the code vision.py
Image detailhighten_shot.yamlNot stated in the paper
Shots per label, k0, 1, 3, 5, 10§ Methods (Prompting)Code also has 20-shot PCam configs; no results reported for them
Prompt development data10 random tiles per label; an initial 30-sample CRC100K set; a separate later 30-sample set§ Methods (Prompting)Development samples may be reused as shots
Test set size60 per binary task; 120 for CRC100K multiclass§ Methods (Datasets)Balanced 1:1 across labels
Input resolutionCRC100K 224x224 px at 0.5 µm/px; PCam 96x96 px at 0.243 µm/px; MHIST 224x224 px§ Methods (Datasets)Resize before sending not stated
Embedding modelPhikon ViT-B (40M, pan-cancer), 768-dimensional§ Methods (kNN-based few-shot image sampling) run_finetune.ipynbTeacher backbone per the paper; extraction notebook not reviewed
Similaritycosinealg. 1 knn_dataset.pyTop-k per label
Text embedding modelOpenAI Ada 002, default settings§ Methods (GPT-4V Model Specifications)Used only for the t-SNE analysis
Baseline optimiserAdam, learning rate 0.001§ Methods (Tile-Level Classification Benchmarks) train_classifier.ipynbBatch size and augmentation not stated
Baseline epochs10§ Methods (Tile-Level Classification Benchmarks) train_classifier.ipynbStopping rule not stated
Baseline losscross-entropy§ Methods (Tile-Level Classification Benchmarks) train_classifier.ipynbUnweighted in the notebook; paper says label distribution is balanced
Baseline initialisationImageNet weights§ Methods (Tile-Level Classification Benchmarks) train_classifier.ipynbExact weight set not stated
Bootstrap100,000 resamples, 2.5% and 97.5% pointsfig. 2 fig. 3Test-set bootstrap only

Results

Author-reported. Setup: GPT-4V in-context learning versus ImageNet-initialised classifiers retrained on the same images for each target, binary tasks with n=60 test images and CRC100K multiclass with n=120 § Methods (Datasets) § Results (Retrained classifiers).

SettingMetricBaselineThis paperSource
CRC100K tumour versus normal, zero-shotaccuracy61.7% (CI 0.500 to 0.733), zero-shot66.7% three-shot, 78.3% five-shot, 90.0% ten-shot (CI 0.817 to 0.967)fig. 2 § Results (ICL improves accuracy)
MHIST, zero-shot vs ten-shot kNNaccuracy56.7% zero-shot83.3% ten-shot kNN (CI 0.733 to 0.917)tab. 1 fig. 2
PatchCamelyon, zero-shot vs ten-shot kNNaccuracy60.0% zero-shot88.3% ten-shot kNN (CI 0.800 to 0.950)tab. 1 fig. 2
MHIST, ten-shot vs best retrained modelaccuracy80.0% Tiny ViT, ten-shot83.3% GPT-4V, ten-shot kNNtab. 1 § Results (Retrained classifiers)
PatchCamelyon, ten-shot vs best retrained modelaccuracy81.7% Tiny ViT, ten-shot88.3% GPT-4V, ten-shot kNNtab. 1 § Results (Retrained classifiers)
CRC100K multiclass (8 labels), five-shotaccuracy96.7% Tiny ViT, five-shot77.5% GPT-4V, five-shottab. 1 § Results (Multilabel CRC100K)
Virchow comparison (cited by the paper)accuracy82.7% (MHIST), 92.7% (PatchCamelyon), as reported in the cited work83.3% and 88.3% for GPT-4V§ Discussion

Full accuracy table (tab. 1, accuracy only; the paper's 95% interval rows are in the source table):

ModelCRC ZeroCRC OneCRC ThreeCRC FiveMHIST ZeroMHIST ThreeMHIST FiveMHIST TenPCam ZeroPCam ThreePCam FivePCam Ten
ResNet-180.1500.7660.8750.9080.5000.6580.8000.7330.5340.7500.7500.717
ResNet-500.0750.7500.8670.9000.4500.5130.6330.6500.5000.5670.7500.550
Tiny ViT0.1420.8500.9500.9670.4670.7500.7170.8000.3670.7330.8000.817
Small ViT0.1670.1500.1330.1830.4500.4740.5000.5660.4830.5830.5500.467
GPT-4V0.3250.6080.7250.7750.5670.6580.7160.8330.6000.7500.8170.883

Note: the caption lists One and Ten for all tasks, but the CRC columns have no Ten and the MHIST columns have no One. The columns are as printed. tab. 1

Further results, in words:

  • Few-shot scaling (binary tasks). More shots improve GPT-4V on every binary task, and kNN selection beats random selection on MHIST and PatchCamelyon fig. 2 § Results (ICL improves accuracy).
  • Zero-shot failure modes. On PatchCamelyon, zero-shot GPT-4V flags every tumour tile but with 80% false positives. On MHIST, it gets most sessile serrated adenomas right (83%) but only a small share of hyperplastic polyps (30%) fig. 3 § Results (Retrained classifiers).
  • Scaling on MHIST and PCam. Normal lymph-node recognition on PatchCamelyon rises from 67% (three-shot) to 77% (five-shot) and 80% (ten-shot). Hyperplastic polyp recognition on MHIST rises from 30% to close to 90% fig. 3 § Results (Retrained classifiers).
  • Multiclass CRC100K. GPT-4V recognises tumour and muscle (recall 80% and 100% in zero-shot), but fails on debris, adipose tissue, lymphocytes, mucus and stroma. Debris is labelled tumour in 93% of cases and stroma muscle in 87%. The best result is five-shot kNN with 40 example images in total. The retrained classifiers stay ahead on this task fig. 4 § Results (Multilabel CRC100K) tab. 1.
  • Text-level reasoning. Text embeddings of GPT-4V's reasoning form clearer clusters per label as the number of shots grows. This is a qualitative analysis with no reported accuracy fig. 5 § Results (Text reasoning).
  • Limitations stated by the authors. Small test sets because of API rate limits, no ensembling (the paper says MedPrompt and Med-PaLM 2 use 44 model calls for a single task), and reliance on a separate pathology encoder for retrieval. The authors also say GPT-4V's training data is unknown, so test-set exposure cannot be excluded § Discussion.

Paper vs code

The repository at code_commit is published under the account Dyke-F, which matches the first author's name, and its README cites this paper. I treat it as the authors' own repository on that basis; the account-to-author link is an inference. It contains the experiment runner, the prompts and the configuration files. The README points to a separate train_classifiers branch for the Phikon and UNI probing scripts; I did not read that branch. I read the main branch only, including the classifier notebook under VisionModels/.

  • Temperature. Paper: 0.1 § Methods (GPT-4V Model Specifications). Code: the API call hard-codes temperature=0.1 vision.py. Every config file sets temperature: 1 ten_shot.yaml, but the call never reads them, because model_kwargs is not passed through vision.py. The effective value is 0.1, and the config values are dead settings.
  • JSON output. Paper: outputs in JSON format § Methods (Prompting). Config: response_format: json_object ten_shot.yaml. Code: the response_format argument is commented out vision.py, so JSON is requested only by the prompt.
  • Patient-level exclusion. Paper: tile embeddings from the same patient are removed "if this information was available" § Methods (kNN-based few-shot image sampling). Code: no patient filter exists in any .py file. The only trace is a commented-out unique_identifier line in the MHIST kNN config ten_shot.yaml. The leakage guard in the code removes only the target image itself knn_dataset.py.
  • Ordering of shots. Paper: the most similar image for each label is shown first § Methods (kNN-based few-shot image sampling). Code: CRC100K and PCam kNN configs keep the most similar first ten_shot.yaml, but MHIST kNN sets most_similar_last: True, which puts the most similar last ten_shot.yaml. Whether the reported MHIST numbers used that setting is not stated.
  • Random sampling. Paper: random few-shot sampling is a comparison arm § Methods (Prompting). Code, MHIST and PCam random configs: the take_random branch shuffles the vectors but then reads names from the unshuffled list, so it returns the first k pool images of each label in file order knn_dataset.py. Code, CRC100K complete configs (no embedding file): pandas.sample, which is not seeded dataset.py. Which code produced the reported random numbers is not stated.
  • Classifier training. Paper: "linear probing ... training a classification layer atop" the four models § Methods (Tile-Level Classification Benchmarks). Notebook: timm.create_model(..., pretrained=True, num_classes=...), with the Adam optimiser given all model parameters train_classifier.ipynb train_classifier.ipynb. This is full fine-tuning, not a linear probe. A separate notebook does train a linear head on frozen Phikon features run_finetune.ipynb. Which one produced Table 1 is not stated.
  • Shot counts. Paper: k in {1, 3, 5, 10} § Methods (Prompting). Code: PCam configs also include 20-shot runs, with no reported results.
  • Failure handling. The __call__ method calls prepare_for_save with a missing argument vision.py. The path is used only when verbose is false, and every config sets verbose: True, so the bug does not fire in the shipped configs.
  • Image MIME type. Tiles are PNG files, but the data URLs are labelled image/jpeg vision.py. The paper does not mention this.
  • Model version. The repository README records gpt-4-vision-preview and says a change of backend is a new experiment, not a rerun. The paper gives the dates of use.
  • Published version. The README cites a Nature Communications article (2024, DOI 10.1038/s41467-024-51465-9). This card covers arXiv v1 only, and I did not check the published article for changed numbers or methods.
  • Code availability. The paper says code will follow publication § Methods (Code availability). The repository is public. No LICENSE file is present, so only paraphrase its content here, with no snippets.

Searches for code. The paper names no URL. A GitHub repository search for "gpt-4v pathology" returned no repositories. A search for "in-context learning histopathology" returned Dyke-F/GPT-4V-In-Context-Learning and an unrelated report-generation project (casperliuliuliu/PathGenIC), which I did not use. The first hit is the repository I cloned.

Pitfalls

  • Temperature is not as stated in the configs. Implement 0.1, as the paper and the call do. The config files suggest 1 and are not used.
  • JSON parsing is on you. The code does not enforce JSON mode, and the paper does not say how malformed answers were counted. Plan a parser and a failure count.
  • Patient leakage. The paper says same-patient tiles are removed when identity is known, but the code does not do it. If patient IDs exist for your data, add the filter yourself. Reference images may come from the same patient as the target, and the paper says this is allowed § Methods (kNN-based few-shot image sampling).
  • "Random" is not reliably random on two tasks. On MHIST and PCam the random arm returns file-ordered examples knn_dataset.py, so its numbers reflect the file order. Use a seeded random draw if you need a true random baseline.
  • Order changes the answer. The paper says most-similar-first; MHIST's config does the reverse. Test both orders before comparing across tasks.
  • Shot ordering across labels. The interleaving stops at the length of the first label's list, so longer lists for other labels are silently cut short vision.py. Keep the lists equal in length.
  • Separator line. The few-shot prompt file must contain a line of eleven hyphens to split the user prompt vision.py. A prompt without it fails with an index error.
  • Paper figures disagree. MHIST ten-shot kNN is 83.4% in the Results text § Results (ICL improves accuracy) and 83.3% in its later text and Table 1 § Results (Retrained classifiers) tab. 1. Use 83.3% (matches the table and the Discussion).
  • Claim about three-shot PatchCamelyon. The text says three- and five-shot prompting outperform all other models on PatchCamelyon § Results (Retrained classifiers). In Table 1, three-shot GPT-4V (0.750) ties ResNet-18 (0.750). Only five- and ten-shot clearly lead.
  • Typo in the model name. The text says "ResNet15" once § Results (Retrained classifiers); the Methods and Table 1 give ResNet-50.
  • Near-chance baseline. The Small ViT values on CRC100K (0.133 to 0.183) are close to the 1/8 rate expected from eight labels. The paper does not discuss this.
  • Test-set size. Sample sizes are 60 and 120. A difference of a few points is inside the reported intervals (for example MHIST ten-shot GPT-4V CI 0.733 to 0.917).
  • Test-set easing. MHIST test images are restricted to unanimous labels, which the authors say makes the task easier § Discussion.
  • Possible memorisation. The authors cannot rule out that GPT-4V saw the test images in training. They argue that zero-shot results near chance make this unlikely § Discussion.
  • Broken data link. The PatchCamelyon source URL in the data-availability statement is malformed (github.com-/basveeling/pcam) § Methods (Data availability). Use the official PCam source.
  • Cost. A CRC100K prompt can carry up to 80 example images (ten shots for each of eight labels) plus the target. The authors note the preview API's request limits § Discussion. Budget per experiment before running the full set.
  • Baseline cost. The retrained baselines need one model per test image (3,600 in total) § Methods (Tile-Level Classification Benchmarks). They are fair comparisons, but expensive to repeat.

Lineage

  • Builds on: MedPrompt, a GPT-4 prompting ensemble with kNN few-shot selection, and in-context learning with text-only models § Introduction § Supplementary Methods (Related Work - Enhancing LLM strategies). Phikon, the pathology encoder used for retrieval, is cited as an embedding source § Methods (kNN-based few-shot image sampling).
  • Compared against: ImageNet-initialised ResNet-18, ResNet-50, Tiny ViT and Small ViT, each retrained for every target image § Methods (Tile-Level Classification Benchmarks); zero-shot GPT-4V as the control.
  • Superseded by: not stated.

Source and license

Adapted from “In-context learning enables multimodal large language models to classify cancer pathology images” by Dyke Ferber, Georg Wölflein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. El Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn, Jakob Nikolas Kather (arXiv:2403.07407), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/Dyke-F/GPT-4V-In-Context-Learning at commit c04b676b2de8 (not stated (no LICENSE file in the repository)). Any quoted code is shown under that license.