Towards artificial general intelligence via a multimodal foundation model

In plain words

The paper trains one large model to match pictures with the text that goes with them, and then checks what it can do with that alone. Most earlier systems of this kind were trained on pairs where the caption describes the picture closely. This one was trained on pairs taken from the web, where a photo and its title may only loosely relate, where a caption may be a loose, mood-level remark rather than a description. The authors argue that this looser matching pushes the model toward more abstract understanding.

The model has two separate encoders, one for images and one for text, that map each input to a point in a shared space. Matching pairs should land close together and mismatched ones far apart. Two design choices keep the cost down: the image side looks at a fixed grid of image patches instead of running a separate object detector, and the model keeps a memory of recent examples so each training step can compare against many non-matching items without needing a huge batch.

Beyond the usual retrieval and classification tests, the authors ask the model to draw what it "thinks" a sentence means, by adjusting an image until it matches the sentence's point in the shared space. They report that the drawings look plausible. The results come from the authors' own experiments, and the work is still a research prototype.

TL;DR

• BriVL pairs two encoders (image and text) and trains them with a cross-modal contrastive loss over 650M weakly correlated Chinese image-text pairs §Results: Pre-Training Data Collection.
• Image features come from an EfficientNet-B7 backbone with multi-scale patch pooling and a self-attention block, not from an object detector §Methods: Image Encoder.
• Negatives come from two momentum-updated queues, so the batch can stay at 2,688 while the queue holds 13,440 entries §Methods: Contrastive Loss §Methods: Implementation Details.
• Author-reported results beat CLIP on zero-shot remote-sensing and news tasks and show gains from pre-training on retrieval and VQA. All numbers come from the authors' own setup, and the released model and code are non-commercial and no-derivatives.

Problem

Most multimodal pre-training models assume that an image and its text describe each other closely (strong semantic correlation), and they match regions to words, often with an object detector and a single network that reads image and text together §Introduction. The authors argue that this assumption limits generalisation, that such data is scarce, and that single-network cross-modal scoring is slow at retrieval time because every query-candidate pair must be run through the model §Introduction. They also note that the detector step is costly, citing a Faster R-CNN cost of 0.064 s per image §Introduction.

Method

BriVL is a two-encoder (two-tower) model trained with a MoCo-style momentum mechanism. The paper describes it as four towers in total, because two momentum encoders are added to maintain negative queues §Methods: Architecture Overview.

Pre-training objective.

1. Each training pair gives an image embedding z_i and a text embedding z_t of dimension d, produced by the trainable encoders §Methods: Architecture Overview.
2. Two momentum copies of the encoders produce positives and negatives. Their weights follow the trainable weights with a slow moving average §Methods: Contrastive Loss eq. θ_m.
3. Two queues hold recent momentum-encoder outputs: N_q image negatives and N_q text negatives. At each iteration the current batch's keys are pushed in and the same number of oldest keys are removed §Methods: Contrastive Loss.
4. For each image, an InfoNCE loss scores its own paired text (positive) against all texts in the text queue, using a dot product divided by temperature τ §Methods: Contrastive Loss eq. L_i2t. The symmetric text-to-image loss uses the image queue eq. L_t2i.
5. The total loss is the plain sum of the two terms, with no weighting eq. L_total.

Encoders.

• Image encoder: a CNN backbone (EfficientNet-B7 in the main model) gives a feature map. Multi-scale patch pooling pools it over a 1×1 grid and a 6×6 grid, giving 37 patch vectors §Methods: Image Encoder. A self-attention block of Transformer encoder layers mixes the patches eq. S', eq. S. Average pooling over the patches, then a two-layer MLP with ReLU, gives the d-dimensional image embedding eq. r^(i).
• Text encoder: a pre-trained Transformer (RoBERTa-Large in the main model) gives one vector per token. The same self-attention block and a two-layer ReLU MLP then produce the text embedding §Methods: Text Encoder.
• Joint embedding size d = 2,560 §Methods: Implementation Details.

Retrieval. At test time the score for a query is the dot product of the query's output with the candidates' outputs §Methods: Contrastive Loss.

Network visualisation (Algorithm 1 alg. 1). The text encoder is frozen. A random 600×600×3 image is optimised by gradient descent on the loss −cos(z_image, z_text), with all other weights frozen. No extra module or data is used §Results: Neural Network Visualization §Methods: Formalization of Neural Network Visualization. Neuron visualisation adds a term that maximises chosen channels of the last layer before pooling; the paper names channels LLP-108, LLP-456 and LLP-678 §Results: Neural Network Visualization.

Text-to-image generation (Algorithm 2 alg. 2). A VQGAN generator pre-trained on ILSVRC-2012 is frozen. Its input is a 16×16 grid of codes from a 1,024-entry codebook of dimension 256. The grid is optimised with the same cosine loss, and after every gradient step each cell is snapped to its nearest codebook entry eq. vq. The same procedure with CLIP is the comparison §Results: Text-to-Image Generation.

Hyperparameters and compute. See the table below. Pre-training used 14 machines with 8 A100 GPUs each (112 GPUs) for about 10 days on DeepSpeed §Methods: Implementation Details.

# One pre-training step, MoCo-style contrast  [§Methods: Contrastive Loss]
# Input:  image batch (Nb, 3, 600, 600); token batch (Nb, l)
# Output: scalar loss; queues updated in place

def training_step(images, tokens, image_queue, text_queue):
    # Trainable encoders give the queries  [§Methods: Architecture Overview]
    z_image = image_encoder(images)                 # (Nb, d)
    z_text = text_encoder(tokens)                   # (Nb, d)
    # Momentum encoders give the positives and the new negatives  [§Methods]
    key_image = momentum_image_encoder(images)      # (Nb, d), no gradient
    key_text = momentum_text_encoder(tokens)        # (Nb, d), no gradient
    # Image-to-text InfoNCE over the text queue  [eq. L_i2t]
    loss_i2t = info_nce(z_image, key_text, text_queue, tau=0.07)
    # Text-to-image InfoNCE over the image queue  [eq. L_t2i]
    loss_t2i = info_nce(z_text, key_image, image_queue, tau=0.07)
    # Plain sum of both directions  [eq. L_total]
    loss = loss_i2t + loss_t2i
    # Push current keys, drop the same number of oldest keys  [§Methods]
    image_queue = push_and_pop(image_queue, key_image)
    text_queue = push_and_pop(text_queue, key_text)
    return loss


# Momentum update of the key encoders after each step  [eq. θ_m]
def momentum_update(m=0.99):
    # Keys follow the trainable weights as a slow moving average
    theta_m = m * theta_m + (1 - m) * theta

# Neural network visualisation  [alg. 1] [eq. L_vis]
# Input:  text; learning rate lr; iteration count (not stated)
# Output: image (600, 600, 3) that the frozen model matches to the text

def visualise_text(text, lr, max_iter):
    # Embed the text once with the frozen text encoder  [alg. 1, line 1]
    text_emb = text_encoder(text)                   # (d,)
    # Start from noise; pixels are the only trainable variables
    image = random_init((600, 600, 3))              # init not stated
    for step in range(max_iter):
        image_emb = image_encoder(image)            # (d,)
        loss = -cosine(image_emb, text_emb)         # [eq. L_vis]
        image = image - lr * grad(loss, image)      # gradient descent
    return image

# Text-to-image generation through a frozen VQGAN  [alg. 2] [eq. L_t2i]
# Input:  text; codebook C (1024 entries of dim 256); generator g
# Output: image generated from the quantised 16×16 code grid

def generate_from_text(text, lr, max_iter):
    # Embed the text once  [alg. 2, line 1]
    text_emb = text_encoder(text)                   # (d,)
    # The code grid U is the only learned matrix  [§Methods]
    codes = random_init((16, 16, 256))              # init not stated
    for step in range(max_iter):
        image = generator(codes)                    # g(U), frozen
        image_emb = image_encoder(image)            # (d,)
        loss = -cosine(image_emb, text_emb)         # [eq. L_t2i]
        codes = codes - lr * grad(loss, codes)      # gradient step on U
        # Snap every cell to its nearest codebook entry  [eq. vq]
        codes = nearest_codebook_entry(codes, C)
    return generator(codes)

Hyperparameters

Results

Author-reported. All numbers come from figure tables in the paper, which are reproduced here with their figure citation. The main pre-training run uses the full WSCD. Zero-shot results use class names only.

Zero-shot remote sensing scene classification, accuracy (%) fig. 4a (UCM). Columns are unseen/seen class ratios. Brackets give standard deviation over 25 random splits, shown only where seen classes exist.

Zero-shot remote sensing scene classification, accuracy (%) fig. 4b (AID).

Zero-shot news classification, accuracy (%) fig. 5a. Title-only input, text encoder only.

Per category on Toutiao News fig. 5b: BriVL w/ RoBERTa-large gains on 10 of 15 categories (largest: sports +88.22, entertainment +60.36) and loses on 5 (largest: automobile −10.58). The paper does not give a single overall gain figure beyond the table above.

Chinese cross-modal retrieval on AIC-ICC, Recall (%) fig. 6a. Test set: first 10K validation images with 50K captions.

Visual7W VQA, "Telling" questions, accuracy (%) fig. 6b. Random 70/30 split.

English tasks, BriVL-en fig. S3a, fig. S3b. Visual7W overall accuracy: 76.91 (direct training), 52.75 (pre-train, zero-shot), 81.26 (pre-train & finetune). Flickr30K test captioning (1K images), direct training vs pre-train & finetune: BLEU@4 18.87 vs 20.18, METEOR 14.44 vs 21.90, CIDEr 37.06 vs 44.47, SPICE 12.71 vs 15.54, ROUGE-L 44.01 vs 45.85.

Ablation, 22M pre-training pairs, 11K test set fig. S2a. Recall (%) sums over six metrics: BriVL 268.66; BriVL without self-attention layers 261.83; BriVL with SimCLR-style in-batch negatives 259.45. BYOL-style training collapsed in the authors' attempts and is not reported.

Qualitative. Visualisations of abstract concepts, sentences and neurons fig. 2a–d, text-to-image examples compared with CLIP fig. 3a–d, and VQA and captioning examples fig. 6c, fig. S3c are shown in the paper without numeric scores. The paper's remote-sensing "baseball field viewed from above" visualisation fig. 4c is also qualitative.

Paper vs code

Official code: the paper's Code availability section links https://github.com/neilfei/brivl-nmi and states the pre-trained model and inference code are under CC BY-NC-ND 4.0 §Code availability. The repository exists, and its latest commit is 6dcdc70 (2022-06-02) according to the GitHub API. The repository's own LICENSE file was not checked.

Because that licence is non-commercial and no-derivatives, the code and weights were not cloned, read, or compared with the paper. No discrepancies were checked. The card is written from the paper alone. Anyone who needs code-level detail must check the repository and its licence before use.

Pitfalls

• Numbering. The source has no equation numbers and its sections are unnumbered. Equations are cited by their symbol, for example eq. L_i2t, and sections by name. Figures and algorithms are cited by the numbers the compiled paper uses in its text (Fig. 1–6, Fig. S1–S3, Alg. 1–2).
• Normalisation is unstated. The InfoNCE loss and the test-time score use a dot product, while the visualisation loss uses cosine. The paper does not say whether embeddings are L2-normalised before the dot product. With τ = 0.07, the scale of the embeddings matters, so an implementation should choose and document one option.
• Temperature. τ = 0.07 is fixed and "selected heuristically". Whether it is learned is not stated, and no sweep is reported.
• Queue ordering. The paper pushes the current keys into the queue but does not say whether the loss is computed before or after the push, or whether the current positive is in the queue at the time of the loss. Each choice changes the denominator.
• Two batch regimes. The main run uses a total batch of 2,688 with queues of 13,440 (5×). The ablation uses 1,728 with queues of 10,368 (6×) on 9 machines and 22M pairs. Ablation results do not transfer directly to the 650M run.
• Momentum and gradients. The paper says momentum encoders and queues produce no gradients. The momentum-update equation is standard, with m = 0.99.
• Learning schedule and length. The learning rate, schedule, warmup and number of steps or epochs are not stated for pre-training or for any fine-tuning.
• Fine-tuning settings. Downstream fine-tuning learning rates, epochs and loss are not stated. The "Fix BN" and unfixed-block choices differ by task, and the best choice in one table is not the best in another.
• Visualisation is underspecified. The iteration count, learning rate, initial noise distribution, optimiser and any image regularisation are not stated. The visualisations are qualitative, and the paper gives no quantitative test of them.
• Text-to-image quantisation. Algorithm 2 snaps the grid to the codebook at every iteration, not only at the end. The text describes quantisation after each update. Learning rate, iterations and initialisation are not stated.
• Zero-shot comparison is confounded. BriVL is given Chinese class names, which were translated from English, while CLIP uses the English names. The paper notes translation and culture effects. No prompt template is given for either model.
• Latency claim without numbers. The paper says the two-tower design is efficient at inference, but reports no latency or throughput measurement for BriVL itself. The 0.064 s figure is for Faster R-CNN, not for BriVL.
• Different small models. The news experiments with RoBERTa-base use a smaller BriVL (EfficientNet-B5, RoBERTa-base) trained on 22M pairs. Do not compare that row with the headline BriVL w/ RoBERTa-large row.
• Data. WSCD (about 650M pairs) is crawled Chinese web data. The filtering beyond pornographic or sensitive content is not described. The paper gives a URL for the pre-training dataset, which was not checked. Downstream datasets (UCM, AID, THUCNews, Toutiao, AIC-ICC, Visual7W) are cited with URLs in the paper.
• Licences. The released model and inference code are CC BY-NC-ND 4.0 according to the paper. Using them commercially or deriving from them is not allowed. The paper's own images are mostly Pexels stock photos §Image Sources. This card uses none of them.
• Ambiguous claim. The paper says BriVL's imagination "even hints of common sense" on VQA and captioning. Those examples are single cases, not measured results.

Lineage

• Builds on: MoCo (momentum encoder and queue for contrastive learning), applied to the cross-modal setting §Methods: Contrastive Loss. Uses InfoNCE and the Transformer self-attention block as in the paper.
• Compared against: CLIP and ALIGN (two-tower, in-batch negatives, strong-correlation data); SimCLR and BYOL in the ablation §Introduction §Ablation Study; BERT, GPT, ViT and BiT as single-modal pre-training §Methods: Architecture Overview; UNITER, OSCAR and M6 as single-tower multimodal models §Methods: Architecture Overview.
• Visualisation builds on Feature Visualization (FeaVis) for CNNs §Results: Neural Network Visualization, and text-to-image uses VQGAN §Methods: Formalization of Text-to-Image Generation.
• Superseded by: not stated in the paper.