Identifying beneficial task relations for multi-task learning in deep neural networks

In plain words

Training one network on several language tasks at once, called multi-task learning, sometimes helps each task and sometimes hurts it. This paper asks which pairs of tasks help each other. Each pair has a main task, which is scored, and an extra task that is only used during training.

The authors train one shared recurrent network on ten tagging tasks, one task at a time, and then run it on all 90 ordered pairs. For each pair they record whether the main task got better or worse. The useful signal comes from how fast each task's training loss falls and where it levels off when the task is trained alone. A simple, readable classifier uses these curve features and some dataset statistics to predict gains. Its best reported result is right in about four cases out of five, well above the majority baseline. The study fixes every setting to the single-task values, so the result answers "does this pair help under these settings" and not "what is the best setting for this pair".

TL;DR

• One bi-directional LSTM with a shared, trainable 100-d embedding layer is trained on ten sequence-labelling datasets, with one dense output layer per task §3.1.
• Each training step draws one task at random and updates from one batch of 32 sentences of that task §3.1.
• For every ordered pair (main, auxiliary), the multi-task model is compared with the main task's single-task model. The label is whether main-task F1 went up §4 fig. 1.
• Logistic regression on 42 features per pair (main-task, auxiliary-task and ratio values of 14 features each) predicts the gain or loss. The authors report 0.785 accuracy and 0.713 F1 tab. 3.
• Settings are the same for single-task and multi-task runs, so the study only says whether a pair helps under those settings §3.1 §4.

Problem

Multi-task learning (MTL) in NLP gives mixed results, and there were no clear conditions under which it helps §1. Earlier work offers a few guesses: auxiliary data should not outweigh the target data §2, MTL may help most when target data is small §2, and success may depend on how evenly the auxiliary labels are spread §2 or on how similar the datasets are §2. The paper tests when MTL helps, not how to make it help, with single-task hyperparameters reused unchanged for the multi-task model, so the predictions are feasible §1.

Method

Shared model §3.1. Each task has its own output layer over a shared encoder. The encoder is a bi-directional LSTM with one hidden layer, shared across tasks. Its inputs are word vectors initialised from pretrained GloVe and updated during training. The embedding parameters are shared too. The model does not distinguish main from auxiliary tasks §3.1.

Training step §3.1. A training step draws one task uniformly at random. It samples a random batch of 32 examples from that task's training data and updates the shared parameters and that task's projection. Each step touches exactly one task. Adadelta is the optimizer §3.1.

# Multi-task training step  [§3.1] [code: rnn_mtl/src/experiment.py#L76-L107]
# Input:  task_list (10 tasks), batch_size (32)
# Output: updated shared weights and one task head

def train_step(task_list, batch_size):
    # Draw one task uniformly; each step trains exactly one task  [§3.1]
    task = uniform_choice(task_list)
    # Sample sentences from that task's training split  [§3.1]
    tokens, labels, lengths = sample_batch(task, batch_size)
    # Forward pass: shared encoder, then the task's own output layer
    states = shared_encoder(tokens, lengths)     # (T, B, 200)
    probs = task_head[task](states)              # (T, B, n_labels)
    # Masked loss: average per sentence, then over the batch
    loss = sentence_mean_loss(probs, labels, lengths)
    # Adadelta update, via this task's own optimizer instance
    optimizer[task].update(loss)

# Shared encoder  [§3.1] [code: rnn_mtl/src/mtl_model.py#L40-L43, #L280-L285]
# Input:  tokens (T, B) of word ids; lengths (B)

def shared_encoder(tokens, lengths):
    # Trainable word vectors, initialised from 100-d GloVe  [§3.1]
    vectors = embedding[tokens]                    # (T, B, 100)
    # One bi-directional LSTM with 100 units per direction  [§3.1]
    states = bilstm(vectors, lengths, units=100)   # (T, B, 200)
    # Dropout on the output, rate 0.1  [code: rnn_mtl/run.sh#L7]
    return dropout(states, rate=0.1)

# Loss used by the code  [code: rnn_mtl/src/mtl_model.py#L123-L134]
# Not stated in the paper; this is what the code computes.

def sentence_mean_loss(probs, labels, lengths):
    # Per-token cross-entropy, zeroed on padding positions
    token_loss = cross_entropy(probs, labels) * (labels != PAD)
    # Sum over the sentence, divide by its length, then average the batch
    per_sentence = sum_over_tokens(token_loss) / lengths
    return mean(per_sentence)

Features for prediction §4. For each of the 10 tasks, the authors compute 14 features from its single-task model and its training data:

1. Seven dataset features: number of training sentences, number of labels, type/token ratio, percentage of training words missing from GloVe, label entropy, Frobenius norm of the sentence-by-term count matrix, and Jensen-Shannon divergence between train and test bags of words §4 tab. 2.
2. Five loss-curve gradients, taken at 10, 20, 30, 50 and 70 percent of the training log §4.
3. Two parameters from a logarithmic fit to the loss curve, L(i) = a · ln(c · i + d) + b; the features are a and c §4.

For each ordered pair (main, auxiliary), the paper uses the main task's features, the auxiliary task's features and their ratio. That gives 3 × 14 = 42 values per pair §4. All features are rescaled to the interval [0, 1] §4.

# Pair row for predicting gain or loss  [§4]
# Input:  features[task] (14 values each), main, aux
# Output: 42 values for one ordered pair

def pair_row(features, main, aux):
    row = {}
    for name in features[main]:
        # Main value, auxiliary value and their ratio for each feature
        row["main_" + name] = features[main][name]
        row["aux_" + name] = features[aux][name]
        row["main/aux_" + name] = features[main][name] / features[aux][name]
    return row

# Curve features from the logged single-task training loss  [§4]
# Code: experiments.ipynb#L195-L215 (grads at 10/20/30/50/70 % of the log)

def curve_features(losses):
    # Finite-difference slope at each fraction of the log  [§4]
    grads = [slope_at(losses, p) for p in (0.1, 0.2, 0.3, 0.5, 0.7)]
    # Fit loss = p0 + p1 * log(p2 * i + p3); the paper's a and c are p1, p2
    p0, p1, p2, p3 = fit_log_curve(losses)
    return grads + [p1, p2]

Labels and classifier §4 fig. 1 tab. 3. For each ordered pair, the gain is the relative change in the main task's micro-averaged F1, from its MTL model against its single-task model, in percent. Pairs with a positive gain are the positive class (40 of 90 in the paper). A logistic regression is trained on the sign of the gain. The reported scores are the mean over 100 runs of randomised 5-fold cross-validation tab. 3.

Single-task and MTL runs §4 fig. 1. Single-task models are trained for 25,000 batches. Multi-task models get 50,000 batches, to offset the fact that each step touches one of the two tasks §4. The code reads F1 at 50,000 and 100,000 batches instead experiments.ipynb; see Paper vs code.

Tasks §3.2 tab. 1. The ten datasets are CCG supertagging, chunking, sentence compression, FrameNet frame and target labelling, POS tagging on tweets, hyperlink prediction, SemEval 2017 keyphrase detection, multiword expressions (STREUSLE), supersense tagging on SemCor, and joint bracket and supersense tagging on STREUSLE §3.2. Dataset statistics, as reported tab. 1:

The last column is single-task micro-averaged test F1 tab. 1.

Hyperparameters

Values marked "paper" are stated in the paper. Values marked "code" are not stated in the paper and come from the released scripts, which the paper does not mention. The paper's full setup is given as "not stated" where neither source gives a value.

Results

Author-reported. Setup: ten sequence-labelling tasks; a shared 100-unit bi-LSTM; 90 ordered pairs; 100 repeats of 5-fold cross-validation of the gain-prediction classifier §4 tab. 3.

Pair-level effects: the paper says chunking and the two supersense tasks contribute most as auxiliaries, and hyperlink prediction does not significantly improve any other task §4 fig. 1. Across the 90 pairs, 40 show a gain §4. In the notebook's own output the largest gains are main hyperlink prediction with auxiliary SemCor (about 23 %) and with auxiliary POS (about 22 %) experiments.ipynb.

Averaged logistic coefficients with the largest magnitudes tab. 4: the main task's curve gradient at 30 % (−1.566), the main task's curve gradient at 20 % (−1.164), the main task's log-curve parameter c (1.007), the main task's number of labels (0.828), and the auxiliary task's label entropy (0.798). Table 4 covers all 42 features §4.2 tab. 4.

The authors' main reading is that multi-task gains are more likely when the main task's loss has flattened early, while the auxiliary task's loss is still falling §4.2.

Paper vs code

• Training length. Paper: single-task 25,000 batches, MTL 50,000 §4. Code: run_batch.sh runs single tasks for 50,000 run_batch.sh, and the logs end at 50,000 for ccg loss_tr.log and at 100,000 for the ccg_com pair loss_tr.log. The notebook reads 50,000 and 100,000 experiments.ipynb. The code is twice the paper's length for both.
• Hidden size. Paper: "a single hidden layer of 100 dimensions" §3.1. Code: 100 units in each direction, so the output is 200 mtl_model.py.
• Dropout. Not in the paper. Code applies dropout 0.1 to the layer output, with no training/evaluation switch run.sh mtl_model.py.
• Learning rate. Not in the paper. run.sh sets 0.1 run.sh; the parser default is 0.01 experiment.py.
• Dropout flag. run.sh passes --dropout, but the parser declares --dropout_rate run.sh experiment.py. Whether the 0.1 reached the model depends on docopt's prefix matching; I did not run it.
• Evaluation split. Paper: F1 on test data tab. 1. Code: the "dev" entries in data.cfg are the _test.conll files data.cfg. Consistent with the paper's table, but the gains and training curves use the test split, not a separate validation split.
• Gain metric. The paper uses micro-averaged F1 fig. 1. The code reads the F1 column for gains experiments.ipynb, although get_f1's docstring says accuracy. The F1 uses positive classes only, excluding the negative label o and the special symbols experiment.py.
• Jensen-Shannon divergence. Paper: between train and test bags of words tab. 2. Code: the test side is built from the label vectorizer, not the input vectorizer. xs_dev is overwritten with the output-vectorized dev data experiments.ipynb; the train side uses input ids experiments.ipynb. So the JSD feature does not compare what the paper describes.
• Majority baseline. My arithmetic from the 40 gains and 50 losses in the notebook output experiments.ipynb, not a stated result: always predicting "loss" gives accuracy 50/90 ≈ 0.556 (the paper prints 0.555), and always predicting "gain" gives F1 = 2·40/(2·40 + 50) = 0.615. So the baseline row pairs the accuracy of one constant classifier with the F1 of the other.
• Ablation rows. The ablation sets features to zero in place on X_abl = X, so each row includes the zeroing of all earlier rows experiments.ipynb. The "best combination" row is not identified in the notebook.
• Undefined helper. The gain loop calls get_performance, which is not defined in the notebook as saved experiments.ipynb.
• Task names. The paper's POS is gpos in the code experiments.ipynb; KEY is keysem17; STR is streusle; SEM is semcor; the other names are the same. data.cfg also lists keyacl and upos, which are not among the ten tasks data.cfg.
• Layers. Task configs register each task at layer 3 task_ccg.cfg, while the model has one layer. The code then uses the top layer and prints a warning mtl_model.py.
• No discrepancy found in: 100-d GloVe, batch 32, 10 tasks, 90 ordered pairs, 14 features per task, 42 per pair, and the curve points at 10, 20, 30, 50, 70 % §3.1 §4 experiment.py experiments.ipynb.

Pitfalls

• Truncation to 30 tokens. Sentences are padded or cut to 30 positions, which include the GO and EOS markers dio.py dio.py. Words beyond that are dropped from both training and evaluation. Evaluation scores positions 1 to 28 of each row experiment.py, so the last words of long sentences are not scored. The paper does not mention a cap.
• Dropout at evaluation time. The dropout op sits in the graph with no mode switch, so the same dropout runs when evaluating and decoding mtl_model.py. The source comments this as a TODO.
• One optimizer per task. Each task gets its own Adadelta instance, so the shared weights carry a separate accumulator per task mtl_model.py mtl_model.py. The paper says only "using Adadelta" §3.1.
• Hidden size wording. "100 dimensions" is per direction in the code, so the layer output is 200, not 100 mtl_model.py.
• Loss is not in the paper. The per-sentence average and then batch average is only in the code mtl_model.py.
• No validation split and no early stopping. Gains are read at fixed iterations on the test split data.cfg experiments.ipynb.
• Randomness is not controlled. Task choice uses Python's random without a seed experiment.py; cross-validation splits use np.random.randint without a seed experiments.ipynb. Training-seed variance is not reported. The "100 runs" is over CV splits only tab. 3.
• Scaling before CV. Min-max scaling is fitted on all 90 rows before cross-validation experiments.ipynb, so the test folds influence the scaling.
• Learning-curve features use training loss. Each logged value is the mean of the task's batch losses since the previous log line experiment.py experiment.py. The gradients are finite differences between neighbouring log lines, not per-iteration values experiments.ipynb.
• Curve fit uses a fixed slice. The fit uses every tenth log line up to index 5,000 experiments.ipynb. For a 50,000-batch single-task log, that is 500 points covering the whole run, but not for other lengths.
• Data is not in the repository. data/ is absent from the clone, so the splits and the conversion from the original corpora cannot be checked here.
• Code is GPL-3.0. The repository is GPL-3.0 LICENSE. Only the pseudocode above is ours, paraphrased; nothing is quoted.
• Sensitive coefficients. Many features are correlated, and the coefficients are averages over random splits experiments.ipynb. The paper's own discussion stresses the curve features, but the coefficient ranking can change with the split.

Lineage

• Builds on: hard parameter sharing in deep networks [Caruana, 1993, as cited in §3]; recurrent MTL for sequence labelling [Søgaard and Goldberg 2016; Klerke et al. 2016; Plank 2016; Alonso and Plank 2017; as cited in §3.1]. The paper reuses hyperparameters across tasks §3.1.
• Compared against: single-task models with the same settings §4. The paper discusses earlier findings from Luong et al. 2016, Hovy et al. 2017, Alonso and Plank 2017, and Mou et al. 2016 §2.
• Superseded by: not known from content in this wiki.