Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

In plain words

Most vision models are built for one job: one model names objects, another finds them with boxes, another writes captions. Florence-2 tries to do all of these with a single model. You give it a picture and a short text instruction such as "describe this" or "find the dog", and it writes its answer as text. When the answer is a location, the location is written as a short sequence of special tokens, so the model never needs a separate output head for boxes or masks.

The model needed a very large amount of training data with many kinds of labels (captions, boxes, phrases tied to boxes). Building that by hand would be slow, so the authors used other models to label the images and then retrained their own model on those labels in several rounds. The result is a compact model that the authors report works well without task-specific training on many tasks.

TL;DR

Florence-2 is a prompt-driven vision model: an image and a task prompt go in, and text (optionally with location tokens) comes out, so one set of weights covers captioning, detection, grounding, segmentation and OCR §3. It is pretrained on FLD-5B, 126M images with 5.4B annotations built by a specialist-model data engine with iterative refinement §4 §5. The authors report strong zero-shot results at 0.23B and 0.77B parameters tab. 4. Check the pitfalls before reusing the numbers: the data engine's exact procedure and several training details are not stated.

Problem

Vision models are usually trained for one family of tasks with task-specific heads. The authors argue that a useful vision foundation model needs both spatial granularity (image-level, region-level and pixel-level understanding) and semantic granularity (from short labels to long descriptions) §1 §2.1. The binding constraint they name is data: such multitask training needs large, dense, high-quality annotations, which are scarce and expensive to produce by hand §1 §4.

Method

Overview. One sequence-to-sequence model takes an image and a text prompt and generates the response as text. All tasks are cast as translation from (image, prompt) to response, trained under one language-modeling loss §3 fig. 2. The task list used in pretraining is in app. A.

Prompts and outputs. Plain-text prompts and answers are kept as text. Region answers use location tokens: the tokenizer vocabulary is extended with 1,000 quantized coordinate bins §3. Three layouts are defined:

• box (x0, y0, x1, y1): top-left and bottom-right corners, used for detection and dense region captioning §3;
• quad box (x0, y0, ..., x3, y3): four corners, clockwise from top-left, used for text detection and recognition §3;
• polygon (x0, y0, ..., xn, yn): vertices in clockwise order, used for referring segmentation §3.

Vision encoder. A DaViT backbone turns an image of size H x W x 3 into N_v visual tokens of width D_v §3. The paper gives no further DaViT detail there; the per-size configuration is in app. C.

Multi-modality encoder-decoder. Prompt tokens are embedded with the extended tokenizer and word-embedding layer §3. The visual tokens go through a linear projection and LayerNorm to the decoder width D, and are concatenated before the prompt embeddings to form the encoder input §3. A standard transformer encoder-decoder processes this input and generates the response text §3.

Objective. Cross-entropy over target tokens, summed over the target sequence eq. 1. The same loss is used for every task §3.

# Florence-2 training step  [§3] [eq. 1]
# Input:  images (B, 3, H, W); prompts and targets: lists of token ids
# Output: scalar loss

def training_step(images, prompts, targets):
    # Encode the image into N_v visual tokens of width D_v  [§3, vision encoder]
    visual_tokens = davit(images)                 # (B, N_v, D_v)
    # Project to the decoder width D and normalise  [§3]
    visual_embeds = layer_norm(linear(visual_tokens))  # (B, N_v, D)
    # Embed the task prompt with the extended tokenizer  [§3]
    prompt_embeds = word_embedding(prompts)       # (B, N_t, D)
    # Concatenate visual and prompt embeddings as encoder input  [§3]
    encoder_input = concat(visual_embeds, prompt_embeds, dim=1)
    encoder_states = encoder(encoder_input)
    # Score each target token given earlier target tokens  [§3] [eq. 1]
    # The decoder input layout is not stated; y_<i implies teacher forcing.
    logits = decoder(targets, encoder_states)     # (B, T, vocab)
    # Sum negative log-likelihood over target tokens  [eq. 1]
    loss = -sum(log_softmax(logits) at targets)
    return loss

Data engine. The data engine has three phases §4.2 fig. 3:

1. Initial annotation. Specialist models, a mix of offline models and cloud services, label each image. Where public data already has annotations (for example Object365 boxes), the existing labels are merged with the synthetic ones §4.2.
2. Filtering. Caption text is parsed with spaCy; texts with too many objects are dropped, and texts must reach a minimum object and action complexity measured on the dependency tree §4.2. Boxes below a confidence threshold are removed and non-maximum suppression is applied §4.2.
3. Iterative refinement. A multitask model is trained on the filtered labels, applied back to the training images, and its predictions are merged into the labels for another round §4.2. The number of rounds is not stated.

Tasks that lacked data for a specialist (for example detailed captions) were left out at phase 1 and labelled later by the refined model §4.2. The paper's account of the annotation pipeline per annotation type:

• Brief text uses a caption specialist, trained on public caption datasets and refined iteratively §4.3.
• Detailed and more-detailed text is produced by large language or multimodal models from existing annotations, but only for a small set; these samples fine- tune the caption specialist, which then labels the rest §4.3.
• Region-text pairs use Azure AI OCR for text regions and a DINO detector trained on public data for objects. Each region gets three candidate texts (object-category phrase, brief caption of the crop, noun chunks of the brief caption), and Florence-1 picks the most similar one §4.3.
• Text-phrase-region triplets use Grounding DINO to ground noun phrases from each text, and SAM to turn boxes into masks. A confidence threshold and a blacklist of pronouns and abstract phrases filter the result §4.3.

Pretraining. Training is under AdamW with cosine decay, DeepSpeed and mixed precision, initialized from UniCL (image encoder) and BART (encoder-decoder) §6.1. Details are in §6.1 and in Hyperparameters below.

Hyperparameters

Results

Author-reported. Setup: one pretrained model per size (base 0.23B, large 0.77B) pretrained on FLD-5B, evaluated zero-shot with no task-specific training §6.2 tab. 4. Baselines are Flamingo (80B) and Kosmos-2 (1.6B), both as reported by the authors.

Generalist fine-tuning (one model fine-tuned on public data, no task-specific heads) §6.3:

Downstream, with DaViT-B (about 80M backbone) §6.4:

Base-size comparison, COCO Mask R-CNN and DINO, and ADE20K UperNet tab. 7 tab. 9: DaViT-B with Florence-2 pretraining reports 53.6 AP_b (Mask R-CNN, 1x schedule) against ConvNeXt v2-B FCMAE at 52.9 (3x schedule), 59.2 AP (DINO) against ViT-B MAE at 55.0, and 54.9 mIoU (UperNet) against BEiT ViT-B at 53.6 (single-scale).

Ablations, all zero-shot tab. 10 tab. 11 tab. 12:

Paper vs code

No official code.

Searched: the paper text, the arXiv abstract page (no code, project page or comments links; only third-party aggregator entries) and the GitHub repository search API for Florence-2. The search returned no repository owned by the authors' organisation for the model; a direct lookup of microsoft/Florence-2 returned 404. The only Microsoft-owned match was microsoft/dstoolkit-finetuning-florence-2 (MIT), a fine-tuning accelerator rather than the model's training code, so it was not used. No code was cloned and no code citations are given.

Pitfalls

• Data engine is under-specified. The paper does not give confidence thresholds, NMS settings, complexity cut-offs, the number of refinement rounds or the stopping rule §4.2. The filtering steps cannot be reproduced exactly from this card; implementers must choose their own values and say so.
• Teacher forcing and decoder input are not stated. eq. 1 conditions on y_<i, which implies teacher forcing, but the paper does not describe how decoder inputs are built or how position information is encoded for the visual and prompt tokens §3.
• Loss normalisation. eq. 1 sums token log-likelihoods. The paper does not say how the sum is averaged over the batch or over tokens, which changes the effective learning rate.
• Location token quantization. The 1,000-bin scheme is stated §3, but the mapping from pixel coordinates to bins (rounding, handling of the image border, and whether bins are per axis) is not.
• Two-stage resolution schedule. Warm-up and LR for the 768 x 768 high-resolution stage are not stated §6.1; only the sample counts are.
• Parameter counts disagree by rounding. Text gives 232M and 771M §6.1; app. C sums to 230M and 770M.
• TextVQA number in the text does not match the table. The text reports 81.5 for TextVQA without OCR tokens and calls it a new state of the art §6.3. The large-model value in tab. 5 is 73.5. Use the table value and check against the PDF.
• Possible typo in the table. In tab. 6 the base model's RefCOCOg test value (82.2) is far below its val value (89.8) and the large model's test value (91.7); verify against the PDF before quoting it.
• Freezing the vision encoder. The text says freezing "does not affect" image-level tasks §6.5. In tab. 12, the frozen-encoder row has higher caption CIDEr (120.0) than the unfrozen default (118.7). The text's claim holds in spirit but the table shows a small gain, not a neutral result.
• Zero-shot improvements over Kosmos-2 are rounded. The text claims about 8 points on RefCOCOg §6.2; the table difference for the large model is 68.0 - 60.6 = 7.4 on val.
• Table and figure numbers are derived. Table and figure numbers in this card follow the order of captions in the LaTeX source, not the rendered PDF. Confirm against the PDF before citing a number in another card.
• Data scale and cost. Labels come from several specialist models including cloud services §4.2. Re-creating the dataset requires access to those services and models. The paper does not state the cost.
• Dataset licence. FLD-5B is built from third-party datasets (ImageNet-22k, Object365, Open Images, Conceptual Captions, LAION) §4.1. The paper does not state the licence of FLD-5B itself, and its annotations were produced by models whose licences are not stated. Do not assume the dataset is reusable.
• Downstream numbers use frozen and unfrozen backbones. The frozen rows in tab. 8 are not the default setting. Compare like with like.

Lineage

• Builds on: DaViT as the vision encoder §3; BART as the language-side initialization §6.1; UniCL as the vision initialization §6.1.
• Uses in its data engine: Florence-1 for matching region texts §4.3; Grounding DINO and SAM for text-phrase-region labels §4.3; DINO for object region labels §4.3. These are named in the paper; this card does not summarize their content.
• Compared against: Flamingo, Kosmos-2, PaLI and PaLI-X, PolyFormer, UNINEXT, Unified-IO, UniTAB, BEiT and ConvNeXt v2 tab. 4 tab. 5 tab. 6 tab. 9.
• Superseded by: not known from content in this wiki.