A Survey of Domain Adaptation for Neural Machine Translation

In plain words

A translation model learned from general text often does poorly on a narrow field such as patents or spoken language, and specialised parallel text is scarce. This survey collects the ways researchers have tried to make neural translation work better in such a field, using text from other fields or plain text in the target field.

The methods fall into two families. One family changes the data: it adds target-language text, creates fake parallel pairs, or picks the most relevant sentences from a large general corpus. The other changes the model or how it is trained or decoded: it mixes in-domain and general training, adds a language model, or adds a domain signal. The authors do not name a single winner. They explain the trade-offs and suggest that the right choice depends on the data one has.

TL;DR

• A survey, not a method: it covers domain adaptation for NMT published up to about 2018 and runs no experiments of its own §1 §7.
• Two categories, data-centric and model-centric, used in the survey's own structure §4 fig. 1.
• Strongest practical pointers, all as reported by the cited works: mixed fine tuning with domain tags, and back-translated synthetic data §4.2.1 §5.
• The survey recommends no single technique §5. Pitfalls: its method descriptions are summaries, and it is written for RNN-based NMT §6.1.

Problem

Neural machine translation (NMT) trained on large general parallel corpora performs poorly on a specific domain when in-domain parallel text is scarce or absent §1. Domain adaptation means using out-of-domain parallel text and in-domain monolingual text to improve in-domain translation §1. The survey covers the case where the domain is known. The case where the input domain is unknown is treated separately in §5.

Assumptions stated by the survey: the NMT model is the attention-based encoder-decoder (RNNsearch) of Bahdanau et al. §2. Many of the cited methods depend on that recurrent architecture §6.1.

Method

The paper is a taxonomy, not an algorithm. Its structure is:

• Background. The RNNsearch model. The translation probability and the cross-entropy training loss are given as equations 1 and 2. The decoder state, the attention weights and the output layer are equations 3, 4 and 5 §2 eq. 1–5.
• Domain adaptation for SMT (background, not NMT): data selection by language-model scores, pseudo-parallel generation, and model-level and instance-level interpolation §3.
• Data-centric methods for NMT §4.1:
  • Monolingual target data. Train a language model on it and fuse it with the NMT model; copy target text to the source side; or use it as a multitask signal in the decoder §4.1.1.
  • Synthetic parallel data. Back-translate in-domain target sentences to make synthetic source sentences, then train on the synthetic pairs §4.1.2 fig. 3.
  • Out-of-domain parallel data. Multi-domain training: append a domain tag to each source sentence and oversample the smaller corpus so each domain gets equal weight §4.1.3. Data selection scores out-of-domain sentences by their similarity to the in-domain set, using the NMT encoder's sentence embedding, and can change the selected subset between epochs §4.1.3 fig. 4.
• Model-centric methods for NMT §4.2:
  • Training objective. Instance or cost weighting, fine tuning, mixed fine tuning, and regularisation such as dropout or L2 §4.2.1 fig. 5 fig. 6.
  • Architecture. Deep fusion with an RNN language model trained jointly, a domain discriminator on top of the encoder, and domain tags added to the word embeddings §4.2.2 fig. 7.
  • Decoding. Shallow fusion with a language model, ensembling of the out-of-domain and fine-tuned models, and stack decoding over word lattices §4.2.3 fig. 8.
• Real-world scenarios (unknown input domain): classify the domain of an input sentence first, then translate with the matching model. For NMT, the survey points to domain-tag and feature methods and to retrieval-based fine tuning §5 fig. 9.
• Future directions: newer NMT architectures (CNN, Transformer), domain dictionaries, multilingual and multi-domain training together, and adversarial or domain-generalisation methods §6.

Mixed fine tuning, the method the authors highlight, combines multi-domain training with fine tuning. In outline §4.2.1 fig. 6:

# Mixed fine tuning with domain tags  [§4.2.1] [§4.1.3] [fig. 6]
# Input:  out_corpus, in_corpus (parallel)
# Output: in-domain NMT model

def mixed_fine_tuning(out_corpus, in_corpus):
    # 0. Append a domain tag to each source sentence  [§4.1.3]
    out_tagged = add_domain_tag(out_corpus, domain="out")
    in_tagged = add_domain_tag(in_corpus, domain="in")
    # 1. Train on the out-of-domain corpus until convergence  [§4.2.1]
    model = init_nmt()
    train(model, out_tagged)
    # 2. Mix the corpora, oversampling the in-domain one  [§4.1.3]
    #    (the oversampling factor is not stated)
    mixed = concatenate(out_tagged, oversample(in_tagged))
    # 3. Resume from step 1 on the mix until convergence  [§4.2.1]
    train(model, mixed)
    return model

The sentence-level data selection in §4.1.3 and the instance weighting in §4.2.1 are the two places where the survey describes a per-sentence weight. The paper gives no formula for either.

Hyperparameters

The survey states no training hyperparameters for any method. Learning rates, batch sizes, dropout rates, oversampling factors, stopping criteria and model sizes are all not stated. The only numbers in the text are corpus sizes quoted to motivate the problem §1:

Results

No experiments. The paper reports no new measurements. Every result below is a claim the survey makes about a cited work. These are author-reported in those works, and this card has not checked them against the originals.

Paper vs code

No official code. The paper has no code release, and it is a literature survey, so there is no method implementation to compare against. The arXiv-linked LaTeX source has no repository link. A GitHub repository search for the title words returned no results.

Pitfalls

• Equation 4 has two typos in the source §2 eq. 4. The context sum reuses the index j where it should use i, and the softmax denominator sums over k = 1..m (the target length) although it should range over the n source positions. An implementer should normalise the attention weights over the source positions, which is what the standard model does.
• Summaries, not reproductions. The survey condenses each method. For exact settings, read the original paper of each method. The Results table is the survey's reading of those papers, not a check of them.
• No head-to-head comparison. The cited results come from different language pairs, data sizes and baselines. They do not rank the methods against each other on one benchmark. The survey itself recommends no single method §5.
• Dated and RNN-centric. The survey was written around 2018 and covers the "past two years" before that §1. Most cited methods use the recurrent NMT model, and the survey says research on the newer architectures is still open §6.1.
• Dated claims about coverage. Statements such as "only one work" on instance weighting in NMT §4.2.1 describe the state of the field in 2018 and were not updated.
• Domain labels are assumed. Most methods need the domain of each training sentence and, in the multi-domain case, the domain of each input. Unknown input domains are only briefly covered §5.
• Combinations are mostly untested. The survey says data-centric and model-centric methods are complementary in principle, but that no study combines them §4.1.3. Its one combined result is the back-translation case in §5.
• Vague wording on mixed fine tuning. The convergence criterion and the oversampling factor are not stated §4.2.1 §4.1.3.

Lineage

• Builds on: the attention-based encoder-decoder NMT of Bahdanau et al., used as the reference architecture §2. Statistical domain adaptation methods, for background §3.
• Summarises the authors' earlier work: the multi-domain and mixed fine tuning methods of Chu et al. (2017) §4.1.3 §4.2.1 fig. 6.
• Compared against: the methods of §4 are compared with each other in the text. No new comparison is run.
• Superseded by: none in this wiki.