Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

In plain words

A sentence-embedding model turns a sentence into a list of numbers so that similar sentences end up with similar lists. Most sentence-embedding models were trained on English only, so their numbers for a German or Arabic sentence are not comparable to the English ones. Training a new multilingual model from scratch needs a lot of data and compute.

The method here reuses the English model as a teacher. A new student model reads the English sentence and its translation and is trained to output the teacher's English vector for both. Because the translation is pushed to the same spot as the original, the two languages end up in one shared space. The idea is simple enough to run with modest data and hardware, and the authors report that it works for many languages, including some with little training data.

TL;DR

• Extends a monolingual sentence encoder to new languages by knowledge distillation: a student (XLM-R) is trained with an MSE loss to reproduce the teacher's English vectors for each source sentence and its translation §2 §3.
• Parallel sentence pairs from OPUS-style corpora and bilingual dictionaries are the training data; the English teacher is SBERT fine-tuned on NLI and STS §3.
• Author-reported: the paraphrase-teacher student reaches 84.6 (monolingual) and 83.7 (cross-lingual) average Spearman on STS 2017, above LASER, mUSE and LaBSE tab. 1 tab. 2.
• On BUCC bitext mining the method trails LASER and LaBSE; the paper says this is expected because it targets semantic similarity rather than exact translations §4.2.

Problem

Sentence embeddings are mostly monolingual, usually English, because training data for other languages is scarce §1. Existing multilingual options each have a cost. LASER is trained on a translation task and works for exact translations but less well for similarity between non-identical sentences §1. mUSE needs multi-task training and hard negatives for its translation ranking task, which is complex and expensive §1. Multilingual BERT and XLM-R, used without parallel data, do not produce aligned vector spaces across languages §1 §8.

Method

The method is a teacher-student setup. The teacher is a fixed English model that already has the desired vector properties. The student is trained to imitate it on both sides of a translation pair. The paper calls this multilingual knowledge distillation §1 §2.

1. Teacher. A monolingual model M that maps source-language sentences (English in the experiments) to fixed-size vectors §2. The experiments use an English SBERT model fine-tuned on NLI and STS data as the teacher §3 §4.1.
2. Student. A model M̂ with the teacher's architecture or a different one §2. The experiments use XLM-RoBERTa, because its SentencePiece vocabulary covers many languages, while an English BERT vocabulary would split other languages into short pieces §3.
3. Parallel data. Pairs (s_i, t_i), where t_i is a translation of s_i. The translations may be in different languages §2. The datasets are OPUS corpora plus two bilingual dictionaries §3.
4. Loss. For each pair in a mini-batch, add two squared distances to the teacher's vector of the source sentence: one for the student's vector of the source and one for the student's vector of the translation. Average over the batch. The equation is unnumbered in the paper §2.
5. Training. Mini-batches of 64 pairs, learning rate 2e-5, up to 20 epochs, 10,000 warm-up steps, with MSE on held-out parallel sentences as the development measure §3.
6. Evaluation. Cosine similarity between sentence vectors. STS uses Spearman rank correlation with gold scores §4.1. Bitext retrieval uses a margin score with BUCC-style thresholding §4.2. Tatoeba similarity search uses nearest-neighbour accuracy in both directions §4.3 app. A.

The loss in pseudocode. The teacher vectors are computed once before training, as the script does make_multilingual.py.

# Multilingual distillation loss  [§2]
# Input:  source, target: B parallel sentences, source in the teacher language
# Output: scalar loss

def distillation_loss(source, target, teacher, student):
    # Teacher vector of the source sentence, computed once  [§2]
    teacher_vectors = teacher.encode(source)        # (B, d_teacher)
    # Student vector of the source sentence  [§2]
    source_vectors = student.encode(source)         # (B, d_student)
    # Student vector of the translation  [§2]
    target_vectors = student.encode(target)         # (B, d_student)
    # Squared distance to the teacher vector; the paper does not say whether
    # the squared error is summed or averaged over dimensions  [§2]
    source_term = squared_distance(teacher_vectors, source_vectors)  # (B,)
    target_term = squared_distance(teacher_vectors, target_vectors)  # (B,)
    # Both terms are added per pair, then averaged over the batch  [§2]
    return (source_term + target_term).mean()

Margin score for BUCC bitext mining, following the paper's formula §4.2. The code computes the same quantity bitext_mining_utils.py.

# Ratio-margin score of a candidate pair  [§4.2]
# Input:  x, y: sentence vectors; knn_x, knn_y: k neighbours of each side
# Output: scalar; the pair counts as parallel above a threshold  [§4.2]

def margin_score(x, y, knn_x, knn_y):
    # Cosine similarity of the candidate pair  [§4.2]
    similarity = cosine(x, y)
    # Mean similarity of x to its neighbours in the other language  [§4.2]
    mean_x = mean([cosine(x, z) for z in knn_x])
    # Mean similarity of y to its neighbours in the other language  [§4.2]
    mean_y = mean([cosine(y, z) for z in knn_y])
    # Divide by the average of the two means: margin(a, b) = a / b  [§4.2]
    return similarity / ((mean_x + mean_y) / 2)

Hyperparameters

Values in the paper come first. The example script in the repository uses different values for the same settings, so both are listed.

Results

Author-reported. The student is XLM-R trained on parallel data. Baselines are the systems listed in each table §4.

Setup. STS 2017 monolingual and cross-lingual sets, extended by the authors with EN-DE, EN-FR, EN-IT and EN-NL pairs machine-translated with Google Translate and checked by fluent speakers §4.1. Metric: Spearman rank correlation times 100 tab. 1 tab. 2. BUCC mining task, F1 tab. 3 §4.2. Tatoeba search over up to 1,000 pairs per language, accuracy in both directions tab. 4 §4.3.

tab. 1 Monolingual STS 2017 (Spearman × 100), setup in §4.1:

tab. 2 Cross-lingual STS 2017 (Spearman × 100), setup in §4.1:

tab. 3 BUCC bitext mining (F1), setup in §4.2:

tab. 4 Tatoeba accuracy, both directions, four low-resource languages, setup in §4.3:

tab. 5 Dataset sizes and STS 2017 results for EN-DE and EN-AR, student XLM-R ← SBERT-nli-stsb §5:

tab. 6 Reduced TED2020 training sizes, STS 2017 Spearman, student XLM-R §5:

tab. 7 Korean STSbenchmark test set, Spearman × 100 §6:

tab. 8 Language bias on the joined multilingual STS set, Spearman × 100 §7:

Appendix Tables 9 and 10 give Tatoeba accuracy for every language with and without parallel training data tab. 9 tab. 10 app. A. They are not reproduced here; the paper says accuracy for languages without parallel data is usually low and varies widely app. A.

Paper vs code

The code is the official repository, UKPLab/sentence-transformers, linked from the paper's abstract footnote [abstract]. The repository is owned by the authors' lab. The commit pinned here is the current main at clone time, which is a later library version than the one used for the 2020 experiments. The paper's own experimental scripts are not in this commit, so the comparisons below are with the example script at this commit.

• Teacher. The paper uses bert-base-nli-stsb-mean-tokens §3; the example script uses paraphrase-distilroberta-base-v2 make_multilingual.py.
• Epochs. The paper trains for a maximum of 20 epochs §3; the script uses 5 make_multilingual.py.
• Warm-up. The paper uses 10,000 steps §3; the script passes 0.1 make_multilingual.py. Under the Hugging Face convention a value below 1 is a fraction of training steps; this was not checked against the pinned library version.
• Training data. The paper uses nine OPUS corpora and two dictionaries §3; the script loads only the talks dataset and comments out the others make_multilingual.py.
• Balancing. The paper draws roughly equal numbers of samples from each dataset per mini-batch and repeats smaller ones §3. The library's default multi-dataset sampler is proportional to dataset size sampler.py, and the script does not set another sampler make_multilingual.py.
• Loss reduction. The paper adds two squared terms and averages over the batch §2. The code computes an element-wise MSE for each of the two columns and averages the two values embed_distill.py. Relative to a per-example squared norm, the code's value is 1/(2d) of the paper's sum, where d is the embedding dimension (inferred from the code).
• Student pooling. The paper does not state pooling for the trained student. The script says that a plain transformer gets mean pooling make_multilingual.py.
• Student input length. Not stated in the paper; the script sets 128 word pieces make_multilingual.py.
• STS data. The paper extends STS 2017 with translated EN-DE, EN-FR, EN-IT and EN-NL pairs §4.1. The script loads mteb/sts17-crosslingual-sts make_multilingual.py. The README points to a separate STS2017-extended.zip for the extended pairs README.md. Whether the mteb set matches the paper's extension was not checked.
• Evaluators. The paper reports BUCC and Tatoeba §4.2 §4.3; the script adds a translation-matching accuracy and an MSE evaluator that the paper does not describe make_multilingual.py.
• BUCC. The paper uses LASER's bitext code with the margin score §4.2. This repository adapts LASER's utilities bitext_mining_utils.py. The script uses k = 4 neighbours and a margin of a / b bucc2018.py bucc2018.py. The paper does not state k, so the value is inferred from the code and may not match the 2020 experiments.
• Score formula. The paper's denominator is (1/2k) of the summed neighbour similarities §4.2. The code takes the mean of each side and averages the two means bitext_mining_utils.py. These are equal.
• Checkpoint. The script saves the model after training ends make_multilingual.py and does not set load_best_model_at_end, so the final checkpoint is kept, not the best dev score. The paper does not say which checkpoint it reports.

Pitfalls

• Dimensions must match. The loss is a direct MSE between student and teacher vectors. Without a projection, the student's output dimension must equal the teacher's mse.py embed_distill.py. This is inferred from the code; the paper does not state it.
• Loss scale. The code's loss is smaller than the paper's sum by the factor 1/(2d). Optimizers that are not scale-invariant, and any fixed loss threshold, will behave differently. The optimum is not affected.
• Training versus evaluation. The loss is an MSE on raw vectors §2, but all evaluation uses cosine similarity §4. The paper does not state whether vectors are normalised before the loss. The script calls encode with default settings make_multilingual.py.
• Stopping rule. The paper says training runs for a maximum of 20 epochs and that MSE on held-out pairs is measured §3. It does not say when training stops early or which checkpoint is reported.
• Choice of initialisation. English SBERT is an unsuitable student initialisation because its English vocabulary breaks other languages into short pieces §3. XLM-R is used instead, and its vocabulary covers 100 languages §3.
• Which model is "10 languages". The paper says one model was trained on 50 languages §4 but §5 refers to "the model trained for 10 languages" §5. §7 evaluates ten language combinations of STS 2017 §7. The paper does not make clear which model §5 means.
• BUCC is noisy. The paper found that 57 of 60 false-positive pairs it inspected were valid translations that the dataset had not labelled as parallel, because the Wikipedia part is not guaranteed to be non-parallel §4.2. The authors say BUCC results should be read with care.
• Tatoeba leakage. The paper excludes Tatoeba from its own training, while LASER was trained on Tatoeba, so LASER's scores may be inflated app. A. Languages without parallel training data score low and vary widely app. A.
• Seed variance. About 0.3 points on large datasets, and larger on small ones §3. Treat gaps of a few tenths of a point as within that noise.
• Hidden cost. The paper argues the method needs less hardware [abstract]. It gives no GPU count or training time for its own models. For comparison, it reports that LASER needed about five days on 16 V100 GPUs §8.
• Language bias. Joined-set results show a small, statistically insignificant drop for mUSE and for this method. LASER and LaBSE drop significantly, at p < 0.001 §7 tab. 8.
• Figures. Figure 1 illustrates the training setup fig. 1. Figure 2 shows a PCA of English and Russian sentence vectors, where LaBSE separates the two languages strongly fig. 2 app. A.

Lineage

• Builds on: Sentence-BERT, whose English NLI and STS model is the teacher §1 §3. XLM-R is the student §1 §3. No other wiki cards were checked for links.
• Compared against: LASER, mUSE and LaBSE §4 §8; mBERT and XLM-R with no parallel data §4.
• Superseded by: not stated in the paper.