BioCLIP: A Vision Foundation Model for the Tree of Life

In plain words

Biologists often want a computer to recognise plants, animals, insects or fungi in photos. Today that usually means collecting labelled photos of each species and training a separate model, which is slow and needs machine-learning skill. This paper builds one general-purpose image model for living things instead.

The authors gathered a very large collection of biology photos, about ten million, covering more than 450,000 named kinds of organism. Each photo carries its place in the tree of life, from kingdom down to species. The model starts from a general image-and-text model called CLIP and keeps training on this collection. Each name is written out as one long string that runs down the whole classification path, so closely related species end up with similar text and similar image features. The model can then name organisms it was never trained on, from a name or from a few example photos. On the ten test sets used, it does clearly better than the general models it was compared with, and the authors also show that its internal features group organisms in a way that follows the tree of life.

TL;DR

• Continues CLIP pre-training of a ViT-B/16 on TreeOfLife-10M, a 10.4M-image biology dataset with taxonomic labels for about 454K taxa §2.1 tab. 1 §4.1.
• Each image is paired with its flattened taxonomic name, or with a scientific or common name chosen at random per training step §3.2.
• Author-reported: mean zero-shot top-1 of 39.4% vs 21.9% for CLIP over 10 tasks, and 50.3% vs 33.6% (1-shot) and 68.8% vs 51.5% (5-shot) tab. 4.
• Zero-shot accuracy on the new Rare Species task, whose species were excluded from training, is 38.0% vs 31.8% for CLIP tab. 4 §4.2.
• Training details are in app. D. The paper does not name its optimiser, and the public code uses a smaller global batch than the paper's main run (see Paper vs code).

Problem

Applying computer vision to a biology question means collecting labelled images for the specific taxa, then finding and training a model for that task §1. Labelled biology data is expensive, so zero-shot and few-shot performance matters §1. Existing general-domain models are trained on hundreds of millions of images but do not separate closely related species well §1 fig. 1. Supervised classifiers treat each class as an unrelated symbol and ignore the taxonomy §3.1. Earlier biology image datasets were too small or too narrow for this purpose: iNat21 has 2.7M images of 10K species §1 tab. 1.

Method

Dataset (TreeOfLife-10M). The dataset merges three sources: iNat21 training split, BIOSCAN-1M insect images, and newly downloaded Encyclopedia of Life (EOL) images §2.1 tab. 1. Labels are unified across sources, with a priority order for conflicting taxonomy records app. C.

1. Images and sources. EOL supplies 6.6M images covering 448,910 classes; iNat21 supplies 2.7M images and 10,000 classes; BIOSCAN-1M supplies 1.1M images and 7,831 classes; the total is 10.4M images and 454,103 unique classes tab. 1 §2.1.
2. Label coverage. About 84% of images have full taxonomic labels app. C. About 10% (BIOSCAN) are labelled only to family; 98.6% of BIOSCAN images have a family label, 22.5% a genus and 7.5% a species §2.1, footnote app. C.
3. Canonicalisation. Taxonomy is merged from EOL, ITIS and iNaturalist, with special handling of homonyms (same genus and species name under different higher taxa) §2.2. Priority, highest first: BIOSCAN metadata, then EOL records, then the iNaturalist taxonomy and iNat21 class names; names still unresolved go through the Global Names Resolver app. C.
4. Release. Metadata CSVs and scripts are released, and the paper says the data is under a public-domain waiver only to the extent the primary sources' licences allow §2.3.

Text types (labels as text). Each class is written in one of several forms §3.2 tab. 3:

• Common name: an English vernacular name; one taxon can have several names and one name can cover several taxa §3.2.
• Scientific name: genus plus species §3.2.
• Taxonomic name: the seven ranks from kingdom to species, joined into one string §3.2.
• Scientific + common and taxonomic + common: the taxonomic or scientific name with the common name appended tab. 3.

During training, at each step, each image is paired with a text randomly drawn from its available forms §3.2. The text is inserted into the standard CLIP template, e.g. "a photo of" followed by the name §3.2.

Model and objective. The model starts from OpenAI's CLIP weights. It has a ViT-B/16 image encoder and a 77-token causal text transformer §4.1. Training continues with CLIP's contrastive objective, which raises the similarity of matched image-text pairs and lowers it for the other pairs in the batch §3.1. The authors argue that the autoregressive text encoder encodes the taxonomy: later rank tokens are conditioned on earlier ranks, so higher ranks are shared by related taxa §3.1 fig. 1.

Training procedure. The main model trains for 100 epochs with a cosine learning-rate schedule, on 8 A100-80GB GPUs over 2 nodes with a global batch of 32,768 §4.1 app. D. Ablation models use 1M randomly sampled examples, batch 16,384, on 4 GPUs of one node §4.1.

Pseudocode for the training step. The blocks below restate the paper's method, with code citations where the public implementation was checked.

# Flatten a taxon into one string, root to leaf  [§3.2]
def taxonomic_name(ranks):
    # ranks: kingdom, phylum, class, order, family, genus, species
    return " ".join(ranks)                     # one string per species

# Pick one text form for this step  [§3.2] [code: src/training/train.py#L96]
def pick_caption(captions_by_form):
    # captions_by_form: five strings for the same image
    return random_choice(captions_by_form)     # one form per step

# One contrastive training step  [§3.1] [§4.1]
def training_step(images, captions):
    # encode both modalities into a shared space  [§3.1]
    image_feats = normalize(vision_encoder(images))      # (B, d)
    text_feats = normalize(text_encoder(captions))       # (B, d)
    # CLIP contrastive loss; matched pairs are on the diagonal  [§3.1]
    # The exact loss form is not written out in the paper (see Pitfalls).
    return clip_contrastive_loss(image_feats, text_feats)

Zero-shot classification. Each class name is embedded with the CLIP template, the normalised embeddings are averaged, and the image is assigned to the nearest class §4.1. The public evaluation code averages over the ImageNet prompt-template set rather than a single template zero_shot_iid.py. Defaults: common names for CLIP and OpenCLIP, taxonomic + common names for BioCLIP, and whatever label the dataset provides when the preferred one is missing §4.1.

Few-shot classification. The authors use a nearest-centroid classifier from SimpleShot. For k shots, k examples per class are drawn at random and their features averaged into a centroid. Features are mean-subtracted and L2-normalised, and each test sample takes the label of its nearest centroid. Each setting is repeated five times with different seeds §4.1.

# Nearest-centroid few-shot classifier  [§4.1]
def few_shot_predict(support_feats, support_labels, query_feats):
    # Subtract the mean of the support features  [§4.1]
    # [code: src/evaluation/few_shot.py#L150]
    mean = support_feats.mean(axis=0)
    support = normalize(support_feats - mean)           # (N_s, d)
    query = normalize(query_feats - mean)               # (N_q, d)
    # One centroid per class: the average of its k shots  [§4.1]
    # [code: src/evaluation/few_shot.py#L154]
    centroids = class_mean(support, support_labels)     # (C, d)
    # Predict the label of the nearest centroid  [§4.1]
    return argmin_distance(query, centroids)

Rare Species task. The authors took 400 species from IUCN categories running from Near Threatened to Extinct in the Wild, each represented by at least 30 EOL images. They removed these species from TreeOfLife-10M and used 30 images per species as a test set, 12,000 images in total §4.2 tab. 2. The IUCN classifications used are those current on 13 July 2023 §4.2, footnote.

Intrinsic evaluation. Image features from the iNat21 validation split (not seen in training) are projected with t-SNE, one plot per taxonomic rank, coloured by label §4.6 fig. 3.

Hierarchical baseline. To test the CLIP objective, the authors trained a model with one linear head per taxonomic rank and summed the cross-entropy losses over the seven ranks §4.4 [listing 1, app. D].

Hyperparameters

Values for the main BioCLIP model unless a row says otherwise.

Public launch script for the training run, for comparison train.sh:

Dataset and evaluation tasks

Evaluation tasks. All tasks are top-1 classification tab. 2. Eight tasks come from Meta-Album, plus Birds 525 and the new Rare Species task §4.1.

tab. 2. The "Examples" column is copied as printed in the paper; the paper does not say whether it counts test or all examples.

Results

Author-reported. Setup: ViT-B/16 for all models, top-1 accuracy, 10 tasks §4.1 tab. 4. "iNat21 only" uses the same procedure but iNat21 instead of TreeOfLife-10M.

Zero-shot tab. 4

One-shot tab. 4

Five-shot tab. 4

Bold marks in tab. 4 show the paper's own best per column: DINO on Plankton (one-shot and five-shot), Supervised IN-21K on PlantVillage (one-shot and five-shot), iNat21-only on Birds 525 one-shot. Supervised IN-21K and DINO are vision-only and have no zero-shot row tab. 4 caption.

Standard deviations. Five runs for few-shot app. E. Example: BioCLIP five-shot Rare Species 65.7 ± 0.43, one-shot 44.9 ± 0.73. Some standard deviations reach 2.4 points (BioCLIP one-shot Insects). Zero-shot results are deterministic app. E.

Text types, zero-shot on Rare Species (10% of the dataset, about 1M images, trained with one text form; accuracy on the test form) tab. 5

tab. 5. The last two rows use full datasets for reference. The mean over all tasks for each text-type model is in app. G, where Mixture scores 32.9 and the single forms score between 22.8 and 28.2.

Objective ablation (macro-average over the 10 tasks, models trained on the 1M-sample subset) tab. 6

tab. 6. Zero-shot is not available for the two cross-entropy models §4.4.

Other reported results.

• Generalised zero-shot on Rare Species with 800 labels (400 seen, 400 unseen): CLIP 23.0%, OpenCLIP 18.2%, BioCLIP 26.0% app. H.
• Plant disease tasks (PlantVillage, PlantDoc): the text says BioCLIP's mean one-shot accuracy is 9.1% above its zero-shot accuracy §4.5. The rows of tab. 4 do not give that: for PlantVillage and PlantDoc the BioCLIP means are 26.6 zero-shot and 44.8 one-shot, a gap of about 18 points. See Pitfalls.
• Intrinsic: t-SNE shows BioCLIP separates orders within Insecta and families within Lepidoptera, where CLIP does not fig. 3 §4.6. This is qualitative; the paper reports no number for it.

Headline claims in the abstract and introduction. The paper says BioCLIP beats the baselines "by 16% to 17% absolute" [abstract]. The table deltas are +17.5 (zero-shot), +16.7 (one-shot) and +17.3 (five-shot) tab. 4. The introduction gives 17% zero-shot and 16% few-shot §1.

Paper vs code

Official repository: Imageomics/bioclip at commit fe4bd610759e03a4235dd5ff4651e5c6ed37f41a README.md. The paper's footnote in §1 points to a project page, which links this repository. The owner is the Imageomics organisation. The paper's acknowledgements thank the "Imageomics team" §6, but the paper does not state the authors' affiliations, so the author link is not verified from the paper itself.

Searched: the paper's arXiv abstract and footnote, the project page, and the GitHub organisation repository list. The paper's Zenodo DOI (10.5281/zenodo.10895871) and Hugging Face datasets are not cloned here.

Differences, each checked against the public code:

1. Global batch size. The paper's main run uses 32,768 on 8 GPUs over 2 nodes §4.1 app. D. The public training scripts use --batch-size 4096 on 4 GPUs, which is 16,384 global train.sh train.sh. The 16,384 figure matches the paper's ablation setup, not the main model.
2. Text-type sampling. The paper says a text is sampled at each step §3.2. The code draws one form for the whole batch, seeded by the step number train.py.
3. Few-shot centroid normalisation. The paper says both the centroids and the test features are mean-subtracted and L2-normalised §4.1. The code normalises each support shot, then averages them into a centroid, and does not normalise the centroid again few_shot.py. The mean is taken over the support set few_shot.py.
4. Zero-shot prompt. The paper shows one "standard CLIP template" example §3.2. The code averages normalised embeddings over the ImageNet template set zero_shot_iid.py zero_shot_iid.py.
5. Optimiser and weight decay. The paper does not name the optimiser. The code uses AdamW main.py, excludes gains, biases, norm layers and logit scale from weight decay main.py, and takes ViT defaults of betas (0.9, 0.98) and epsilon 1e-6 when unset params.py.
6. Zero-shot default text. The paper's default for BioCLIP is taxonomic + common names §4.1. The public evaluation script defaults to taxon_com eval_zero_shot.sh.
7. Few-shot repetitions. Paper: five seeds §4.1. Code: --nfold default 5 and --kshot_list default [1, 5] params.py params.py. Consistent.
8. Licence. The GitHub API reports the repository as "Other". The LICENSE file is the standard MIT text, with copyright held by the Imageomics Institute and by the OpenCLIP authors LICENSE. This card records MIT.

No discrepancy was found for the peak learning rate (1e-4), warm-up (1000 steps), weight decay (0.2), cosine schedule or 100 epochs train.sh.

Pitfalls

• Reproducing the headline number. Using the public 4-GPU script gives a global batch of 16,384, half the paper's main run §4.1. The paper does not say whether the main model was trained with the public script. Expect different results if the batch is not matched.
• Text length. The text encoder has a 77-token context §4.1. Long taxonomic plus common strings can approach that limit. The paper does not say how overlong text is truncated, and the card does not check the tokenizer code.
• Contrastive loss form. The paper describes CLIP's objective in words §3.1 but writes no equation for it. Whether the loss is symmetric (image-to-text and text-to-image) and the logit scale or temperature are not stated in the paper. The loss implementation lives in the vendored open_clip module, which this card did not read.
• Per-step vs per-image text. The paper's wording ("at each training step", §3.2) can be read as per-image or per-step. The code is per-batch train.py.
• Few-shot normalisation. The paper says to normalise the centroid. Re-normalising matters because the distance is Euclidean; the public code does not do it (see Paper vs code, item 3). Implementers should decide which they want and say so.
• Epoch numbers in the appendix. The iNat21-only model's 65 epochs, and the 86 to 91 for the 1M-sample models, are the epochs with the lowest validation loss app. D. They are not training budgets.
• Label noise. The authors found after training that some homonyms were mislabelled at higher ranks (family up to kingdom). This affects about 0.1 to 0.2% of data. A fix was promised but is not in the paper app. C.
• Data licence. Images from EOL carry per-image licences. The paper says the dataset release is public-domain-waived only to the extent the primary licences allow §2.3. Anyone reusing the image data, rather than the model code, must check each image's licence.
• Rare Species list is dated. Species come from IUCN categories as of 13 July 2023 §4.2, footnote. Later IUCN updates will change the set.
• Table arithmetic. Recomputing the mean from the printed rows gives 22.0 for CLIP and 39.5 for BioCLIP, against 21.9 and 39.4 printed in tab. 4. This looks like rounding. Use the printed values with that caveat.
• Ablations are not the main model. The text-type and objective ablations train on a 1M-sample subset §4.3 §4.4 and their numbers are not directly comparable to the main table.
• Few-shot variance. Standard deviations over five seeds reach about 2 points on several tasks, up to 2.4 app. E. Differences smaller than that between single runs mean little.
• Plant-disease text claim. The 9.1% one-shot-over-zero-shot figure in §4.5 is not reproduced from tab. 4 (see Other reported results). Quote the table, not the sentence.
• Compute. The main run needed 8 A100-80GB GPUs over 2 nodes for 100 epochs §4.1. The paper gives no wall-clock time.
• Hierarchical baseline. The hierarchical cross-entropy baseline's code is not in this card's check. The paper gives its loss in a listing [listing 1, app. D] only.

Lineage

• Builds on: CLIP, which this paper continues training from OpenAI weights §4.1 (cited in the paper as radford2021learning).
• Compared against: OpenAI CLIP and OpenCLIP trained on LAION-400M §4.1; a supervised ImageNet-21K ViT and DINO for few-shot §4.1.
• Ablation baselines: iNat21-only training tab. 4; cross-entropy and hierarchical cross-entropy tab. 6.
• Superseded by: not stated in this card's source.