RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER

In plain words

Tweets often come with a picture that has nothing to do with the text. Earlier tagging systems that look at the picture pull in whatever parts of it catch their attention, even when the picture is unrelated, and that can push a name into the wrong category. RpBERT first asks a simpler question: does the picture add anything to the tweet? It answers with a single score. The picture's features are then scaled by that score, so an unrelated picture is largely ignored and a relevant one is used more. A standard name tagger then labels the words of the tweet.

TL;DR

RpBERT is a BERT-Base model that reads the tweet text and the image together and is trained on two tasks: whether the image adds meaning to the text, and named entity recognition (NER). The relevance score scales the image features before they reach a biLSTM-CRF tagger § Model Design § Multitask Training for MNER. The paper reports gains over text-only and earlier multimodal taggers on two Twitter NER datasets, but these are author-reported and come from single runs with no variance given § Results of MNER.

Problem

Attention-based multimodal NER models pick image regions from the text query whether or not the image relates to the text, so irrelevant pictures inject misleading visual context § Introduction fig. 1. The authors state that about 56% of instances in the Bloomberg text-image relation dataset are "Image does not add" pairs, and that a classifier they ran on a random sample of Twitter100k marked about 60% as such § Introduction. The relation dataset comes from Vempala and Preoţiuc-Pietro § Datasets. The Twitter100k figure comes from the authors' own classifier, which this card cannot check.

Method

RpBERT is one BERT-Base encoder that accepts text and image tokens in a single sequence. A relevance score r is computed from the [CLS] state of a first pass, and r scales the visual tokens of a second pass that feeds NER.

1. Input sequence. The sequence is [CLS] words [SEP] image blocks. Words are tokenised by the BERT WordPiece tokenizer. The image is resized to 224 x 224, passed through ResNet-152, and its 7 x 7 final grid is read as 49 block regions of dimension 2048. A learned matrix W^v maps each block to the BERT width § Model Design. Segment embeddings mark text (A) and image blocks (B) § Model Design.
2. Relevance score. The [CLS] output goes to a fully connected layer with two outputs. The softmax probability of the positive class is r = pi_1 § Model Design.
3. Gate variants. The paper compares three ways to turn the probabilities into the gate G:
   • soft, G_s = softmax(x) § Relation Propagation;
   • hard, G_h1 = [softmax(x) > 0.5], trained with a straight-through estimator § Relation Propagation;
   • Gumbel-softmax, G_h2 = softmax((x + g) / tau) with Gumbel noise g and a temperature annealed from 1 to 0.1 during training § Relation Propagation.
4. Propagation. The score fills a 49 x d matrix R, and the visual input is R ⊙ V. If r = 0, every visual token is zeroed. The authors describe the score as used to construct R § Model Design.
5. Task 1, relation classification. The TRC data (Bloomberg) is split into binary labels: "Image adds" against the rest. The model is trained with cross entropy on the [CLS] output § Multitask Training for MNER § Datasets.
6. Task 2, NER. A second pass runs the same encoder on the text with R ⊙ V as the image part. The output for each word is the mean of its sub-token outputs. It is concatenated with the word and character inputs of a biLSTM-CRF tagger, and the model minimises the CRF negative log-likelihood. Test-time decoding uses Viterbi § Multitask Training for MNER.
7. Schedule. In each epoch, all TRC batches run first, then all NER batches alg. 1.

# Relevance gate and propagated visual input  [§ Model Design]
# Input: tokens (words), image (224 x 224 x 3)

def relevance_score(tokens, image):
    # One BERT pass over [CLS] words [SEP] 49 image blocks  [§ Model Design]
    hidden = bert_with_image(tokens, visual_tokens(image))
    cls_state = hidden[:, CLS_POS]
    probs = softmax(fc_relation(cls_state))  # (B, 2)
    return probs[:, 1]                       # r = pi_1, (B,)

def visual_tokens(image):
    # 7 x 7 ResNet-152 grid as 49 block embeddings  [§ Model Design]
    grid = resnet152_features(image)         # (B, 2048, 7, 7)
    blocks = flatten(grid)                   # (B, 49, 2048)
    return blocks @ W_v                      # (B, 49, d_bert)

def propagated_visual(image, r):
    # R ⊙ V: every block is multiplied by the same score  [§ Model Design]
    return r[:, None, None] * visual_tokens(image)  # (B, 49, d_bert)

# One epoch of multitask training  [alg. 1]
# Input: TRC batches (text, image, adds_label),
#        NER batches (tokens, tags, image)

def train_epoch(trc_batches, mner_batches):
    # Task 1: image-adds classification on TRC data
    # [§ Multitask Training for MNER]
    for text, image, adds_label in trc_batches:
        cls_state = bert_with_image(text, image)[:, CLS_POS]
        loss_1 = cross_entropy(fc_relation(cls_state), adds_label)
        step(fc_relation, bert, resnet, proj)

    # Task 2: NER with relation propagation  [§ Multitask Training for MNER]
    for tokens, tags, image in mner_batches:
        r = relevance_score(tokens, image)        # gate from the first pass
        visual = propagated_visual(image, r)      # R ⊙ V
        word_feats = bert_with_visual(tokens, visual)  # per-word mean
        emissions = bilstm(concat(word_emb, word_feats))
        loss_2 = crf_neg_log_likelihood(emissions, tags)
        step(bilstm, crf, bert, resnet, proj)     # FCs not listed in alg. 1

# Gate variants for r  [§ Relation Propagation]
# probs: softmax outputs of the relation head, (B, 2)

def gate(probs, variant):
    # Soft gate G_s: keep the probability as is
    if variant == "soft":
        return probs[:, 1]
    # Hard gate G_h1: step at 0.5, softmax gradient (straight-through)
    hard = (probs[:, 1] > 0.5).float()
    return hard + probs[:, 1] - probs[:, 1].detach()

For each image block, the attention analysis averages the attention from the words over layers and heads § Case Study via Attention Visualization. The statistic S_TV averages attention over all layers, heads, words and blocks, and is compared between Snap Res. test pairs with r > 0.5 and r < 0.5 § Ablation Study fig. 3.

Hyperparameters

Values are from the paper's table and text unless marked "code". Where the paper gives a value, the code default is shown in "Paper vs code" below.

Results

Author-reported, single configuration per row. The "Fudan Univ." dataset is the one from Zhang et al. (2018), and "Snap Res." is the one from Lu et al. (2018) § Datasets. F1 is in percent.

Other visual-linguistic baselines, as reported in the paper (F1, Fudan / Snap): Yu et al. 73.4 / 85.3; VL-BERT 72.4 / 86.0; ViLBERT 72.0 / 85.7; UNITER 72.7 / 86.1 § Results of MNER. The paper reports 74.9 and 87.8 as state of the art, 1.5 and 2.5 points above Yu et al. § Results of MNER.

Text-image relation classification (TRC), F1: RpBERT 88.1 against 81.0 for Lu et al. (+7.1) § Result of TRC.

Ablation, soft gate, with and without relation propagation (w/o Rp), F1:

Source: § Ablation Study with the table headed by "Image adds" and "Image doesn't add".

Paper vs code

The official repository at the cited commit is https://github.com/Multimodal-NER/RpBERT. Its LICENSE file is MIT, copyright "SuYindu", and one of the paper's authors is Yindu Su. The paper links this repository in the abstract, so it is the official code. The repository holds a single model and training path (model/model.py, main.py, utils.py, data/). The paper's hard-gate and Gumbel variants are not in the release.

• Gate variants. The paper trains and compares soft, hard (G_h1) and Gumbel (G_h2) gates § Relation Propagation § Results of MNER. The code implements only the softmax probability of the positive class model.py. No straight-through estimator and no Gumbel noise or temperature schedule appear in the repository. The hard-gate and Gumbel rows of the table cannot be reproduced from it.
• Gradient of the NER loss into the relevance head. The paper's algorithm lists the parameters updated by the NER loss as biLSTM, CRF, RpBERT and ResNet, and not the TRC fully connected layers alg. 1. The code computes the gate from the TRC head aux_head without detaching it model.py, so the NER loss also updates that head, and main.py puts it in the optimiser when --gate is set main.py.
• Loss scaling for relation classification. The paper's losses are sums over the data § Multitask Training for MNER. The code averages the relation loss over the batch and multiplies it by 0.05 when it is trained main.py utils.py model.py. The NER loss is a CRF mean over the batch model.py or a per-sentence sum divided by the batch size model.py.
• Learning rates. The paper's table gives 1e-4 for biLSTM-CRF and 1e-6 for RpBERT and ResNet § Settings. The code's default is 1e-5, applied to BERT and ResNet, and 100 times that (1e-3) for the head, LSTM, CRF, projection and relation layers main.py main.py.
• Batch size. The paper gives 8 § Settings. The code default is 16 main.py. The README's commands do not set it.
• Dropout. The paper's table gives 0.5 § Settings. The code defines no dropout layer of its own model.py. Only the dropout inside the Hugging Face BERT and the flair modules is active.
• Gradient clipping, scheduler, weight decay. Not stated in the paper. The code uses none of them main.py.
• Epochs. Not stated in the paper. The code defaults to 10 epochs main.py, and it reports the test F1 from the epoch with the best dev F1 main.py. The paper does not say how its reported epoch was chosen.
• Word embeddings. The paper uses only fastText Crawl 300-d § Settings. With --stacked, which the README's commands use, the code stacks Crawl, GloVe-style Twitter word vectors and two flair contextual embeddings (news forward and backward) model.py README.md. The paper's 25-dimensional character embedding has no counterpart in the code.
• Visual encoder. The paper uses ResNet-152 § Model Design. The README asks for ResNet-101 weights and its command uses --encoder_v resnet101 README.md README.md. main.py accepts both main.py.
• Visual projection. The paper says its fully connected layers are linear followed by ReLU § Settings. The projection of visual features is a plain linear layer, with no activation model.py model.py.
• Visual position embeddings. The paper says visual tokens share one position § Model Design. The code appends the projected image features to the text embeddings as inputs_embeds and passes them to the Hugging Face BERT encoder model.py. Under that API the positional embedding is added to every position, so image tokens probably get distinct positions. This is inferred from the call and not run.
• Image preprocessing. The paper gives the 224 x 224 resize only § Settings. The code also applies ImageNet mean and standard deviation normalisation dataset.py.
• Relation dataset split. The paper uses an 8:2 split § Datasets. The code takes the first 3,576 rows as training data and the rest as test data loader.py loader.py. The total size is not given in the paper, so the split cannot be checked.
• Snap Res. statistics. The paper gives 4,817 / 1,032 / 1,033 tweets § Datasets. The README table lists 4,290 / 1,432 / 1,459 sentences for Twitter-17 README.md. These counts do not match the paper's split.
• Text cleaning. The code strips a trailing t.co link from relation tweets loader.py and replaces empty tokens, special characters and t.co tokens in NER data with [UNK] loader.py. The paper does not mention either step.
• Metric. The paper reports F1 § Settings. The code computes strict IOB2 entity-level F1 with seqeval utils.py.
• Ablation. The paper's "w/o Rp" is Task 2 only § Ablation Study. In the code this matches running with --encoder_v but without --gate, which leaves the visual features unscaled model.py.
• Seeds. The code fixes seed 0 by default and enables deterministic algorithms main.py utils.py. The paper reports one number per setting, with no run-to-run variance.

Pitfalls

• Missing gate variants. Only the soft gate can be reproduced from the code. The hard and Gumbel results in the paper are author-only claims.
• Extra flags are required. --gate uses the relation head from --aux. Without --aux, that head is never trained, so the gate is random. main.py only checks that --encoder_v is set main.py.
• Two BERT passes per NER batch. The gate needs a full forward pass over the text and image before the propagated pass model.py. Training cost is about double per NER batch, and the gate's gradient flows through the second pass.
• Gumbel formula typo. The paper writes the Gumbel-softmax as (x + g) // tau § Relation Propagation. The division symbol is presumably /, but the paper does not say so explicitly.
• Temperature schedule. The paper says the temperature is annealed from 1 to 0.1 without giving the shape of the schedule § Relation Propagation.
• Straight-through details. The hard gate needs the softmax gradient while the forward value is a step. Getting the detach placement wrong gives a hard gate with no training signal to the relation head.
• Attention-shift claim. The paper says the mean of S_TV on irrelevant pairs drops "by 20%" from 0.041 to 0.034 § Ablation Study. That is a drop of about 17%, so the text and the numbers disagree. The increase on relevant pairs, 0.042 to 0.102, is as stated § Ablation Study fig. 3.
• Averaging over subwords. Word features are the mean over sub-token outputs model.py. The paper describes the same step for OOV words § Multitask Training for MNER, but not whether it applies to every word.
• Padding and image position. Image tokens are appended after the padded text model.py model.py. The padding length depends on the batch, so the image positions shift with it. This is an implementation detail the paper does not address.
• Dataset mapping. The paper's names for the two NER datasets are Fudan Univ. and Snap Res., not Twitter-15 and Twitter-17. Use the README's mapping when fetching the data README.md.
• Relation data are not in the repo. The TRC data (Vempala et al.) must be downloaded separately README.md. Its size is not stated in the paper.
• Single run. Results come from one configuration per row with no variance. A difference of a few tenths of a point between rows is not supported by the paper.

Lineage

• Builds on: BERT-Base § Settings and the biLSTM-CRF tagger of Lample et al. § Multitask Training for MNER. Image features come from ResNet § Model Design.
• Compared against: attention-based multimodal NER models (Lu et al. 2018, Zhang et al. 2018, Arshad et al. 2019), visual-linguistic BERT models (VL-BERT, ViLBERT, UNITER) and Yu et al. 2020 § Results of MNER. None of their content is used on this card.
• Superseded by: not known from the content of this wiki.