Multi-task learning

Overview

In multi-task learning, one model is trained on several objectives at once. The usual shape is a shared encoder with one output head or decoder per task, and a total loss that combines the task losses, most often as a weighted sum 1711.08195 §3.6 2101.06679 eq. 5. The weights can be fixed, or adapted during training from gradient norms and loss ratios, as GradNorm-style methods do 2606.02979 §3.3.

Several other routes keep no per-task loss weights. Florence-2 casts every task as text output and trains all of them under one token-level loss, so the task is picked by a prompt rather than by a head 2311.06242 §3 2311.06242 eq. 1. Unified-IO does the same for images, boxes, masks and depth, which all become tokens in one vocabulary 2206.08916 §3.1. MAGVIT uses one masked-token loss, and the task is set by which pixels are given as condition 2212.05199 §3.2. Med-PaLM M fine-tunes one large multimodal language model on a mixture of tasks, also written as instructions with text answers, and sets how often each task is sampled rather than how much each loss counts 2307.14334 §4.2 2307.14334 tab. A.1. VIMA conditions one robot policy on a prompt and trains it with a single imitation loss 2210.03094 §4. PPTOD writes each dialogue sub-task as a text-to-text prompt and trains one T5 model with one token-level likelihood 2109.14739 §3.2 2109.14739 eq. 2. T0 fine-tunes one T5 model on 62 datasets, each written with several prompts, and scores only the target text under one likelihood 2110.08207 §4 2110.08207 §5. Multi-Game Decision Transformer trains one sequence model on tokens from 41 Atari games, with next-token cross-entropy over return, action and reward tokens 2205.15241 §3.1. Painter takes another output form. It writes each dense output as an RGB image and trains one masked-image model with a pixel regression loss, and a solved example placed before the query selects the task 2212.02499 §3.1 2212.02499 §3.3.

A last route keeps the losses but changes how their gradients are merged. CAGrad starts from the average task gradient and picks a nearby update that improves the worst task, with a radius c around the average 2110.14048 §3.1 2110.14048 alg. 1. Anabranch Network is listed here for its second loss, a camouflage classifier that gates a saliency map, but its default training is two-stage, not joint 2105.09451 §5.3.

Twenty-six cards mention multi-task training as a design, and the variants below describe each one. A 2016 survey among them, A Taxonomy of Deep Convolutional Neural Nets for Computer Vision, describes two CNN routes: a multi-task loss with primary and auxiliary tasks, and task-specific sub-networks 1601.06615 §3.5. The cards do not agree on joint versus separate training. MAGVIT reports joint models far ahead of separately trained single-task models on video FVD 2212.05199 §4.4. Elsewhere the ranking changes with the dataset 2110.04994 tab. 6, and the one ablation that adds a planning loss to a detector leaves detection about the same 2101.06679 tab. 3. Med-PaLM M's one ablation removes a task from a finished mixture rather than training a single-task model, and it shows small drops on a related task, from one run each 2307.14334 tab. 6. Anabranch Network's joint variant moves its four FCNs in different directions, and the authors say it does not help consistently 2105.09451 §5.5.2 2105.09451 tab. 6. The camouflage ranking paper gives a joint-versus-single comparison on one dataset. Its joint model beats a camouflage-only model on the COD metrics, with S-measure 0.756 against 0.723, but its ranking-only model has a slightly lower rank MAE, 0.046 against 0.049 2103.04011 tab. 5. The annotator paper compares its multi-task model with a separate model per annotator. On the majority label of the GHC data, the multi-task model scores F1 59.49 against 53.54 for the ensemble, but on GoEmotions it is worse on four of six emotions in the full data 2110.05719 tab. 1 2110.05719 tab. 3. The driving-perception paper compares one compact network with combined single-task baselines. It has the lower total metric on three of four test sets and loses on set C 2606.02979 tab. 2. The shared-encoder driving detector YOLOP shows the same split. Its joint model is level with its detection-only model, and half a point below its drivable-area-only model on mIoU, but on lane lines the lane-only model reaches 79.6 accuracy against 70.5 for the joint model 2108.11250 tab. 5. Its region-based baseline, R-CNNP, drops on detection and segmentation when trained jointly 2108.11250 tab. 6. The multitask NAM beats single-task NAMs on a synthetic set built to favour sharing, with MSE 0.712 against 1.083 2004.13912 tab. 2, but on COMPAS the multitask and single-task intervals overlap 2004.13912 tab. 3. The three newest cards are mixed too. The driving network TCP's shared-backbone model scores 48.27 on the CARLA driving score, against 45.03 for an ensemble of two separate models. The ensemble has about twice the parameters, 46.81M against 23.58M 2206.08129 tab. 4. Painter's joint model beats separately trained models on most dense tasks, but keypoint AP is 72.1 jointly against 72.5 separately, and SIDD SSIM is 0.954 against 0.956 2212.02499 tab. 2. LIQE's three-task model reaches mean SRCC 0.922 over six datasets, against 0.915 for quality alone. Its scene and distortion accuracies fall when quality is added, from 0.908 to 0.894 and from 0.847 to 0.831 2303.14968 tab. 5. CAGrad's NYU-v2 table measures each method against separately trained single-task models, with Δm (relative change, lower is better) of 0.20 for CAGrad and 5.59 for MTAN 2110.14048 tab. 1. The MAGVIT comparison is on video generation, and the others are on different tasks, so the cards are not direct evidence for each other. Text results point the same way, with the same caveat. The rumour model beats its single-task branch LSTM on PHEME and RumourEval 1806.03713 tab. 3, and the sentiment and sarcasm model beats its standalone classifiers by about three average-F points 1901.08014 tab. 1. Both report single runs with no variance. PPTOD's annotation ablation moves in the same direction on end-to-end dialogue 2109.14739 tab. 8. Treat the choice of joint versus separate training as something to test on your own data.

Variants

Auxiliary tag loss on a report generator (the paper's design)

The image encoder, a VGG-19 network, feeds two parts. Its last fully connected layers predict a distribution over medical tags, and its convolutional map gives 196 region features 1711.08195 §3.2. The tag predictions are used in two ways. The tag loss trains the classifier 1711.08195 §3.6. The ten most likely tags are then embedded and attended to, next to the regions, when the report decoder writes each sentence 1711.08195 §3.2 1711.08195 §3.3.

The decoder has its own losses: a stop/continue loss per sentence and a word-level cross-entropy 1711.08195 §3.4 1711.08195 §3.5. All four terms are summed with weight 1.0 each 1711.08195 §4.2. Card: Automatic Generation of Medical Imaging Reports.

Gradients reach the tag classifier only through the tag loss. Whether decoder gradients flow back through the selected tag embeddings is not stated 1711.08195 §3.2.

The paper's attention coverage penalty is a regulariser on the decoder, not a separate task. It has its own weight, also 1.0, and the paper does not say how it is normalised over regions or sentences 1711.08195 §3.6 1711.08195 eq. 9 1711.08195 §4.2.

The paper gives no run without the tag loss, so it does not isolate what the joint training adds 1711.08195 §4.3 1711.08195 tab. 1.

Perception and planning losses with a weight beta

A single CNN takes ten LiDAR sweeps and an HD map and feeds a perception head (anchor classification and regression over future frames) and a cost-volume head 2101.06679 §3.1. The total is perception loss plus beta times planning loss 2101.06679 eq. 5. The planning loss is a max-margin loss over sampled trajectories that reads the cost volume, so it trains that output rather than a separate task head 2101.06679 §3.3 2101.06679 eq. 8. Card: End-to-end Interpretable Neural Motion Planner.

The weights alpha and beta are not stated 2101.06679 eq. 5 2101.06679 eq. 7. The ablation adds planning to a detector without changing the detection metrics much: mAP at IoU 0.5 is 94.1 detection-only and 94.2 with planning, and at IoU 0.7 it is 81.3 and 81.1 2101.06679 tab. 3. Against planning alone, the joint model is worse on collisions at 2 s (0.29 against 0.23) and better on violations at 3 s (3.26 against 3.85) 2101.06679 tab. 3. Each row is one run with no seeds, so differences of this size may be noise 2101.06679 tab. 3.

One token-level loss over many vision tasks

Florence-2 takes an image and a text prompt and writes the answer as text. Boxes and polygons are written as location tokens, with 1,000 quantized bins per coordinate 2311.06242 §3. Every task is trained under the same cross-entropy over target tokens, summed over the target sequence 2311.06242 eq. 1. Because there is one loss, there are no per-task weights to set 2311.06242 §3. Card: Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks.

The model is pretrained on FLD-5B, whose annotations come from specialist models and a refinement loop that retrains the model on its own labels 2311.06242 §4.2. The authors report zero-shot results at 0.23B and 0.77B parameters 2311.06242 tab. 4. The paper's ablations change the amount of data and the vision and language pretraining 2311.06242 tab. 11 2311.06242 tab. 12. None trains the same model on one task only, so the card cannot say how much of the result comes from the shared training.

Shared vocabulary for vision and language outputs (Unified-IO)

Unified-IO turns images, masks, boxes, keypoints and depth maps into discrete tokens and trains one T5-style encoder-decoder on them, with no task-specific heads 2206.08916 §3.1 2206.08916 §3. Text uses subword tokens, points and boxes use 1,000 location tokens, and dense outputs and images use codes from an ImageNet-pretrained VQ-GAN 2206.08916 §3.1 2206.08916 §3.2. Training mixes 95 datasets from 62 sources 2206.08916 §3.3 2206.08916 tab. 1. The eight task groups are sampled at one eighth each, except image synthesis at 3/16 and dense labelling at 1/16. Datasets inside a group are weighted by the square root of their size 2206.08916 §3.3. The paper gives no loss equation, and the learning-rate decay is underspecified 2206.08916 §3.4. Card: Unified-IO.

The ablation holds out one task group at a time, on the LARGE model for 200k to 235k steps, against 250k for the full run 2206.08916 tab. 5 2206.08916 §4.3. Holding out image synthesis drops keypoint score from 69.2 to 18.3, the largest change in the table, while the other holdouts move their scores by smaller amounts. The ablation runs are shorter and use a smaller model than the reported XL model, so the authors warn against reading them as the full model's behaviour 2206.08916 §4.3. The main comparison reports 64.3 on the GRIT test split for the XL model, against 32.0 for GPV-2 2206.08916 tab. 3.

Masked token modelling over ten video tasks (MAGVIT)

MAGVIT compresses a video into a small grid of discrete codes with a 3D vector-quantised autoencoder. A bidirectional transformer is then trained to fill in masked codes 2212.05199 §3.1 2212.05199 §3.2. Ten tasks, such as frame prediction, interpolation, in- and outpainting and class-conditional generation, share the transformer. The paper gives each task its own prompt token and condition shape. The known pixels are tokenised and written into the masked positions, so one token cross-entropy covers every task 2212.05199 §3.2 2212.05199 app. B.1. The loss splits into terms for condition-region positions, masked positions and the remaining positions 2212.05199 eq. 3 2212.05199 eq. 4. Card: MAGVIT.

Joint results on the authors' benchmarks are reported as debiased FVD, where lower is better. On BAIR, one model trained on eight tasks averages 32.8 for the base size and 22.8 for the large one. Single-task models from the same paper average 150.6 and 201.1 2212.05199 §4.4. On Something-Something v2 with ten tasks, the averages are 43.4 and 27.3, against 258.8 and 402.9 for the single-task models 2212.05199 §4.4. These are the authors' numbers, and the paper is the only source for them. See the card for the parts of the public code that reproduce them.

One prompted policy across task templates (VIMA)

A robot policy takes a prompt that interleaves text and object images, plus its recent observations and actions, and predicts the next action 2210.03094 §4. Each detected object becomes one token, the prompt goes through a pretrained T5 encoder, and a causal decoder cross-attends to the prompt 2210.03094 §4 2210.03094 app. C.2.1. Training is behaviour cloning, with one negative log-likelihood over the expert actions and no per-task weights 2210.03094 §4 2210.03094 app. D. The benchmark has 17 task templates in six categories, evaluated at four zero-shot levels 2210.03094 §3. At 200M controller parameters, success on the level 4 novel tasks is 48.6% against 16.8% for the best baseline, VIMA-Gato 2210.03094 tab. 10. The baselines are the authors' own re-implementations 2210.03094 §5.1. The card reports no run trained on one task at a time, so the paper does not measure the gain from joint training directly. Card: VIMA.

Instruction mixture with zero- and few-shot templates (Flan)

The Flan 2022 collection gathers about 1,800 tasks, each rendered with templates, and the paper fine-tunes T5-XL on the mix 2301.13688 §3 2301.13688 §3.3. The paper's task count varies between sections, and the card records the variation under Pitfalls 2301.13688 §3.3. The authors test three choices: mixing zero-shot and few-shot templates [arxiv:2301.13688 §3.2], adding inverted tasks, where the answer becomes the input, at a 30% rate 2301.13688 §3.4 2301.13688 app. B, and balancing the mixture across sources 2301.13688 §3.5. The mixture weights are set by leaving each source out in turn and measuring MMLU, then by judgement. The paper gives no numeric weights 2301.13688 §3.5 2301.13688 app. A.1. For the three sources the card names, the leave-one-out differences are at most about 0.5 MMLU points, and the paper reports no seeds or error bars 2301.13688 tab. 2.

The abstract says mixed templates help "in all settings", but dropping the few-shot templates gives a zero-shot chain-of-thought score of 38.9, against 35.8 for the full mix 2301.13688 tab. 1. Fine-tuning from the Flan model converges faster on a single target task 2301.13688 §4 2301.13688 fig. 5. Card: The Flan Collection.

Prompted multitask fine-tuning, tested on held-out task types (T0)

T0 fine-tunes a T5 encoder-decoder, adapted for language modelling, on 62 English datasets in 12 task types. Each dataset is written as several prompts, and the model is trained only to produce the target text, by maximum likelihood 2110.08207 §3 2110.08207 §4 2110.08207 §5. Four task types are held out entirely: natural language inference, coreference, sentence completion and word sense disambiguation 2110.08207 §3. Large datasets share a 500,000-example budget across their prompts, and the paper's description of this sampling differs from the released code, so see the card. Card: Multitask Prompted Training Enables Zero-Shot Task Generalization.

On the held-out sets, T0 matches or exceeds GPT-3 on 9 of 11 datasets, and Winogrande and HellaSwag are the exceptions (author-reported) [arxiv:2110.08207 §6.1] 2110.08207 §7. The GPT-3 numbers use one prompt per dataset, while the T0 numbers are medians over all its prompts, so the two are not matched 2110.08207 §6.1. Against its own T5+LM starting point, T0 improves on every held-out dataset 2110.08207 §6.1 2110.08207 fig. 4. Moving from 5.7 prompts per dataset to all prompts raises the median on 9 of 11 datasets 2110.08207 §6.2. One T0 run took about 27 TPU v3-512 hours 2110.08207 app. B.1 2110.08207 tab. 1.

Shared recurrent encoder with per-task heads for rumour verification

The rumour model uses hard parameter sharing. One LSTM stack reads each reply branch, and each task then has its own dense layers and output. The paper tests a veracity-plus-stance model, a veracity-plus-detection model and a three-task model 1806.03713 §4.2. Veracity is the main task, and stance and detection are auxiliary 1806.03713 §1 1806.03713 §2.1. Card: All-in-one: Multi-task Learning for Rumour Verification.

A task with no label for a branch drops out of the loss for that branch 1806.03713 §4.2. The paper states a plain sum over task losses. The released code uses fixed weights of 0.3, 0.4 and 0.3, so the stated and implemented weights differ; see the card. Hyperparameters are chosen by a Tree-structured Parzen Estimator search over 30 trials on the development set 1806.03713 §5.1.

Shared GRU with attention pooling per task and a fusion link (sentiment and sarcasm)

One GRU of hidden size 500 is shared by both tasks. Two ReLU projections of size 300 feed attention pooling for each task, and the attention parameters are shared 1901.08014 §2 1901.08014 §4. A neural tensor network fuses the two task vectors into a 100-dimensional vector 1901.08014 §2. The sarcasm head reads its own vector with the fused one. The sentiment head reads only its own vector, because adding the fused vector gave worse results 1901.08014 §2. The two cross-entropy losses have equal priority, and the paper leaves the sum implicit 1901.08014 §2. Card: Sentiment and Sarcasm Classification with Multitask Learning.

On 994 sentences with 10-fold cross-validation, the best variant averages 86.66 F. Standalone classifiers average 83.75, and the fusion model without shared attention averages 86.10 1901.08014 tab. 1. No variance is reported 1901.08014 tab. 1. The cited state of the art uses gaze features that this model ignores 1901.08014 §4.

Eight transform-prediction heads on one 1-D CNN (self-supervised activity recognition)

Pre-training uses only unlabelled accelerometer windows. Each window is kept as it is, with label False, and also copied after one of eight signal transforms, with label True. Each transform has its own binary head 1907.11879 §3.1 1907.11879 alg. 1. The transforms are noise, scaling, sign inversion, negation over a time window, time reversal, segment permutation, time warping and channel shuffling 1907.11879 §3.2. One shared trunk of three 1-D convolutional blocks feeds all eight heads. Each head is a 256-unit dense layer with a sigmoid output 1907.11879 §3.3. The loss is binary cross-entropy per task, summed with task weights, and each batch mixes all tasks 1907.11879 eq. 2 1907.11879 §3.1. The weights are not stated, and the paper lists uncertainty-based weighting as future work 1907.11879 §4.3.3. Card: Multi-task Self-Supervised Learning for Human Activity Detection.

After pre-training, the heads are discarded. The trunk's convolutions feed a new activity classifier, either frozen or with the last convolutional block fine-tuned 1907.11879 §3.1 1907.11879 §4.3.2. The activity labels are used only at that second stage.

Results are author-reported, as Cohen's kappa averaged over 10 runs, on six public smartphone sets. Frozen features beat an autoencoder with the same encoder on all six. On HHAR they score 0.7310 against 0.5994, and on WISDM 0.7986 against 0.5106 1907.11879 tab. 2. Supervised end-to-end training is higher on UniMiB, WISDM and MotionSense, with 0.7815, 0.8211 and 0.8763 against 0.7642, 0.7986 and 0.8589 for frozen features 1907.11879 tab. 2. Under 5-fold user-split cross-validation, frozen features are slightly ahead of supervised training on HHAR, 0.7881 against 0.7788 1907.11879 tab. 4. The "±" values are not defined, and the random seeds are not stated 1907.11879 tab. 2. No official code was found, so the transform parameters and the batch size come from the paper alone, and several of them are not stated there 1907.11879 §3.2 1907.11879 alg. 1.

Text-to-text pre-training across dialogue annotation subsets (PPTOD)

One T5 model handles intent, belief state, dialogue action and response. Each sample is a task prompt, the dialogue context and a target text 2109.14739 §3.2. Pre-training pools eleven corpora, each labelled for only some sub-tasks, and builds samples only from the labels a corpus has 2109.14739 §3.1 2109.14739 tab. 1. Card: Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System.

The annotation ablation runs one annotation type at a time, with a single value per cell and no spread shown. At 1% of MultiWOZ 2.0 end-to-end data, Inform reaches 66.10 with dialogue-state pre-training only, 58.58 with intent only, and 66.96 with all four annotations, against 53.28 with no multi-task pre-training 2109.14739 tab. 8. In low-resource DST, PPTOD-large reaches joint goal accuracy 31.46 at 1% of the data, against 13.21 for SOLOIST 2109.14739 tab. 5.

One sequence model over many games (Multi-Game Decision Transformer)

One decoder-only transformer is trained with next-token cross-entropy over sequences of observation patches, a return token, an action and a reward 2205.15241 §3.1. The sequence formulation lists no task or game token 2205.15241 §3.1. Training data are offline DQN replay trajectories from 41 games 2205.15241 §3.3. At play time the model samples a high target return before each action 2205.15241 §3.4. Card: Multi-Game Decision Transformers.

The author-reported result is 126% of human-normalised IQM across the 41 training games, against 101% for the training dataset 2205.15241 §1 fig. 1. The same model beats behaviour cloning in 31 of 41 games 2205.15241 §4.7. Temporal-difference baselines such as CQL do not improve with size and drop for the larger models 2205.15241 §4.4. Most results come from a single seed; only the 40M model was run over five seeds 2205.15241 app. B.1.

Camouflage classifier gating a saliency map (Anabranch Network)

Anabranch Network runs two streams on the same image. The segmentation stream is any end-to-end saliency FCN; the authors use four, DHS, DSS, SRM and WSS 2105.09451 §4.2 2105.09451 §5.3. The classification stream puts three fully connected layers on the FCN's convolutional features and ends in a two-way softmax, trained with cross-entropy 2105.09451 §4.2 2105.09451 tab. 1. The classifier's probability is one number, and it multiplies every pixel of the segmentation map 2105.09451 §4.2. Card: Anabranch Network for Camouflaged Object Segmentation.

The default training is two-stage. The segmentation stream is fine-tuned on CAMO, then frozen while the classifier is trained on CAMO-COCO 2105.09451 §5.3. The joint variant adds the two losses without weights and trains both streams together 2105.09451 eq. 2 2105.09451 §5.5.2. The paper does not say which layer feeds the classifier, and the hidden width is 4096 in the text but 2048 in the table 2105.09451 §4.2 2105.09451 tab. 1.

The authors' comparison is with each FCN alone. ANet-SRM scores F-beta 0.826 and IoU 0.732 on CAMO-COCO with the adaptive threshold, against 0.699 and 0.535 for the pre-trained SRM 2105.09451 tab. 3. The same SRM fine-tuned on CAMO-COCO scores F-beta 0.830 and IoU 0.717 2105.09451 tab. 3, so the gain over a fine-tuned baseline is mixed. These are single-run numbers on the authors' test sets.

Gradient merging with a conflict-averse radius (CAGrad)

CAGrad keeps one loss per task and changes how the gradients are merged. It starts from the average task gradient g0 and adds a step toward the blend of task gradients that most improves the worst task. The step stays within a radius of c times the norm of g0, with c in [0, 1) 2110.14048 §3.1 2110.14048 eq. 3. At c = 0 the update is plain gradient descent on the average loss, and large c approaches MGDA 2110.14048 §3.1 2110.14048 app. A.1. The blend weights come from a problem over K simplex weights, not from a fixed weight per task 2110.14048 §3.1. Card: Conflict-Averse Gradient Descent for Multi-task Learning.

Each step needs one backward pass per task. CAGrad-Fast samples a subset of tasks and averages the rest, and the paper reports speed-ups of about 2× on Meta-World MT10 and 5× on MT50 against PCGrad 2110.14048 §5.3 2110.14048 eq. 4 2110.14048 tab. 7.

The author-reported results use c = 0.4 on NYU-v2, with Δm 0.20 against 3.97 for PCGrad, 3.58 for GradDrop and 1.38 for MGDA 2110.14048 tab. 1. On CityScapes, Δm is 11.64 against 22.60 for MTAN, but the segmentation mIoU is 75.16 against 75.18 for MTAN 2110.14048 tab. 2. On Meta-World, success is 0.83 on MT10 and 0.52 on MT50, against 0.72 and 0.50 for PCGrad 2110.14048 tab. 3. The supervised runs use three seeds and a p = 0.1 significance mark, so several margins are small 2110.14048 tab. 1 2110.14048 tab. 2.

The convergence theorem covers the plain step with a step size of at most 1/H, but the experiments pass the direction to Adam or SAC, so the guarantee does not carry over directly 2110.14048 §3.2 2110.14048 §5.2.

One head per annotator on a shared encoder (Davani et al.)

The model is one BERT-base encoder with one linear softmax head per annotator. Each head is trained only on the instances its annotator labelled, and the loss is summed over the available labels 2110.05719 §3.4 [arxiv:2110.05719 §4.2]. The final label is the majority over all heads at prediction time 2110.05719 §3.4 2110.05719 §4.3.1. The same heads give an uncertainty score, the variance of the head labels, which the authors use as a disagreement measure 2110.05719 §4.3.2 2110.05719 eq. 1. Card: Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations.

Here the tasks are the annotators, not the labels. The paper's claim rests on the annotator-level labels, so the design needs them for each instance 2110.05719 §6.2. The authors report seven binary tasks, but the card keeps the GHC and GoEmotions results apart, since their setups differ 2110.05719 §4.2. The GHC individual-label F1 is 63.20 for this model, against 60.94 for the per-annotator ensemble 2110.05719 tab. 1. The F1 values are means over five repeats, and the paper reports no significance test 2110.05719 §4.3.1.

Joint fixation and camouflage decoders with reverse attention

One ResNet-50 backbone feeds a fixation decoder, which predicts the region that makes a camouflaged animal visible. Its output, 1 minus the fixation map, guides the features of a second, camouflage decoder 2103.04011 §3.2. The two losses are summed with weight λ = 1 2103.04011 eq. 3. A third task, a Mask R-CNN-style ranking head with a rank loss, is in the paper but not in the released code 2103.04011 §3.3 2103.04011 tab. 5. Card: Simultaneously Localize, Segment and Rank the Camouflaged Objects.

The fixation term is binary cross-entropy in the paper and mean squared error on the sigmoid output in the code, and the segmentation term is a pixel-position-aware loss in the paper and a weighted BCE plus IoU in the code 2103.04011 §3.2 2103.04011 tab. 2. The code's second camouflage output is an extra loss term the paper does not describe. The ranking loss weights are shown only as a figure, with an indexing ambiguity 2103.04011 §3.3 2103.04011 fig. 4.

The ablation trains a fixation-only, a camouflage-only, a ranking-only and the joint model on the same 2,000-image set. Joint training gives S-measure 0.756 for camouflage against 0.723 for camouflage-only. For fixation, the joint model is about level with the fixation-only model on SIM and CC and has a lower KLD, 0.995 against 1.457 2103.04011 tab. 5. Each row is one run, and the paper does not state the seed 2103.04011 tab. 5 2103.04011 §4.1.

Shared-input feature nets with per-task weighted sums (multitask NAM)

Each input column gets several small feature nets, and each task has its own weighted sum over those nets, so every task's shape function is a learned mix of shared parts 2004.13912 §4.2 2004.13912 app. A.8. The task outputs are added and a bias is included 2004.13912 §4.2. The paper's synthetic experiment uses 6 nets per feature for 6 tasks [arxiv:2004.13912 §4.2.1]. Card: Neural Additive Models: Interpretable Machine Learning with Neural Nets.

The gain on synthetic data is measured against single-task NAMs, averaged over 20 runs 2004.13912 tab. 2. The synthetic data were built to favour multitask learning, so the gap says little about real tabular tasks [arxiv:2004.13912 §4.2.1]. On COMPAS the two models' intervals overlap 2004.13912 tab. 3. The authors say the generator code will be released, and the card did not check the separate multitask repository that the README links [arxiv:2004.13912 app. A.8.1].

Per-input encoders, per-output decoders and GradNorm weights (compact driving perception)

One network reads RGB, event-camera and LiDAR inputs, each through its own encoder, and writes four outputs through separate decoders: depth and segmentation for four views, LiDAR segmentation and a bird's-eye-view map. Two shared bottlenecks sit between them 2606.02979 §3.1. The first fuses the input features and feeds the depth, segmentation and LiDAR decoders. The second takes what those tasks extracted and feeds the BEV decoder [arxiv:2606.02979 §3.1]. The loss is a weighted sum of task losses, with the weights adapted by a modified GradNorm 2606.02979 eq. 9 2606.02979 §3.3. The method measures gradient norms on one chosen layer per bottleneck, and updates the weights once per epoch to save compute 2606.02979 eq. 10 2606.02979 §3.3. Card: Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion.

On the simulated CARLA sets and nuScenes-lidarseg, the total metric of the adapted model is lower than that of the combined single-task baselines on three of four test sets, with 1.393 against 1.430 on set A, 1.086 against 1.160 on set B and 1.069 against 1.124 on nuScenes 2606.02979 tab. 2. On set C the baselines win, 0.976 against 0.999 2606.02979 tab. 2. The baselines are the authors' own retrainings, and the paper reports the models use under 2% of the baselines' parameters 2606.02979 tab. 3 2606.02979 §5.4. The adaptive weights lower the total metric against the same network with fixed weights on all test sets 2606.02979 §5.2 2606.02979 tab. 2. The ± values in the table are variances, not standard errors, so they show spread only [arxiv:2606.02979 tab. 2].

Shared encoder with a detector and two segmentation heads (YOLOP)

One CSPDarknet encoder with a spatial pyramid pooling module and a feature pyramid feeds three heads: an anchor-based vehicle detector and two segmentation heads, for drivable area and lane lines 2108.11250 §3 2108.11250 §3.1. Both segmentation heads read the same stride-8 feature map 2108.11250 §3.2.2. The total is a weighted sum of the three task losses, and the paper gives no values for the weights 2108.11250 eq. 4 2108.11250 §3.3. The lane loss adds an IoU term to cross-entropy 2108.11250 eq. 3. Card: YOLOP: You Only Look Once for Panoptic Driving Perception.

The default training is end-to-end 2108.11250 §3.4. The paper also gives a staged schedule: the encoder and detector first, then the segmentation heads with the rest frozen, then all three together, with each stage ending on a loss threshold 2108.11250 alg. 1. On the authors' BDD100K table the staged run is within half a point of end-to-end on detection AP and drivable mIoU, but lower on recall, 87.5 against 89.2, and on lane accuracy, 68.0 against 70.5 2108.11250 tab. 4. The paper reports one run per row, with no seeds.

Its speed claim is for the joint network. Multitask inference takes 24.4 ms per frame, against 15.7, 14.8 and 14.8 ms for three single-task models, which sum to 45.3 ms 2108.11250 tab. 5. The paper says all runs used one GTX TITAN XP 2108.11250 §4.1.3. The Jetson TX2 figure is quoted in the introduction 2108.11250 §1. The released code differs from the paper in several loss details, so see the card before you reimplement the losses.

Trajectory and control branches with a fixed blend (TCP)

One image encoder feeds two output branches for a single front camera. The trajectory branch predicts K waypoints, which two PID controllers turn into throttle, brake and steer. The control branch predicts the current action and K future actions 2206.08129 §3.2. Its per-step attention over the image grid is computed from both branches' hidden states 2206.08129 eq. 2 2206.08129 eq. 3. Both branches imitate an RL expert, and their losses are summed with weights 2206.08129 eq. 6. The supplement sets the trajectory and control weights to 1, and states the auxiliary weights in a sentence the card reads two ways 2206.08129 §3.3. At test time the two controls are blended with weight 0.3, and a turning rule decides which output gets more weight 2206.08129 §3.4. Card: Trajectory-guided Control Prediction for End-to-end Autonomous Driving.

In the ablation, the first row is a control-only model with a driving score of 32.45. Adding the trajectory task gives 34.98, which lies inside the plus-minus values the paper reports. The temporal module, the trajectory-guided attention and the final blend bring the score to 57.01 2206.08129 tab. 3. The paper does not define the plus-minus values, so the size of each gain is hard to judge. All closed-loop results come from one simulator 2206.08129 §4.1.

The released code differs from the paper in several places, and its LICENSE file says Apache-2.0 while the supplement says CC BY-NC 4.0 for the code. The leaderboard entries cannot be reproduced from the repository alone. See the card before you reuse any of it.

One pixel-space model prompted by a solved example (Painter)

Every dense output is turned into an RGB image of the input's size. Depth, segmentation, keypoints and instances each get a colour code, and restoration tasks already are images 2212.02499 §3.1. Training stacks two pairs from the same task into one sample, hides part of one output, and regresses the hidden pixels with a smooth L1 loss 2212.02499 §3.2. A new task needs only new training pairs, with no change to the architecture or loss 2212.02499 §3.1. At inference, a solved pair is placed above the query, and the query's output is painted in 2212.02499 §3.3. Card: Images Speak in Images: A Generalist Painter for In-Context Visual Learning.

The paper's tables compare the joint model with specialists and generalists. Painter's depth RMSE of 0.288 beats the best specialist, 0.330, but it trails the specialists on ADE-20K mIoU, 49.9 against 57.7, and on COCO panoptic PQ, 43.4 against 57.8 2212.02499 tab. 1. On NYUv2 depth, RMSE is 0.291 for a random prompt, 0.288 for a searched one and 0.286 for a learned one 2212.02499 tab. 3. The paper reports no variance, so these gaps may be run-to-run noise.

The paper's training recipe differs from its released script. The paper gives a 75% mask ratio, but the script masks 784 of 1568 patches, which is 50%. The card has the details 2212.02499 §3.2.

Joint quality, scene and distortion heads with dynamic weights (LIQE)

A fine-tuned CLIP image encoder is scored against 495 text labels. Each label combines one of five quality levels, one of nine scene types and one of eleven distortion types 2303.14968 §3.1. The quality, scene and distortion outputs are marginals of one softmax over those labels 2303.14968 §3.2 2303.14968 eq. 3. The quality term is a pairwise fidelity loss, the scene term is a multi-label fidelity loss, and the distortion term is a multi-class fidelity loss 2303.14968 eq. 6 2303.14968 eq. 7 2303.14968 eq. 9. The three weights come from dynamic weight average, a softmax over each loss's descent ratio, with a temperature the paper does not give 2303.14968 eq. 11. Card: Blind Image Quality Assessment via Vision-Language Correspondence.

Quality is the main target. The full model averages SRCC 0.922 over six datasets, against 0.915 for quality alone 2303.14968 tab. 5. The auxiliary tasks add 0.007 to mean SRCC in that table. Equal task weights reach 0.920, and the paper reports no spread for these ablation rows 2303.14968 tab. 4.

The card lists the temperature and the pair-averaging differences from the released code. Its text encoder and label names also differ from the paper, so see the card before you reproduce the weights.

When to use which

Implementation sketch

A shared-trunk step with one weighted loss per task. The variants differ in what the heads read and whether the weights exist at all, as noted under each line.

# Shared-trunk multi-task step  [arxiv:2101.06679 eq. 5] [arxiv:1711.08195 eq. 8]
def multitask_step(batch, trunk, heads, weights):
    # One forward pass through the shared encoder  [arxiv:2101.06679 §3.1]
    features = trunk(batch.inputs)
    total = 0.0
    # One head and one loss per task  [arxiv:1711.08195 §3.6]
    for name, head in heads.items():
        # Some losses read other outputs, e.g. the planning loss reads the
        # cost volume  [arxiv:2101.06679 eq. 8]
        loss = head.loss(features, batch.targets[name])
        # Weighted sum; the weights are a design choice  [arxiv:2101.06679 eq. 5]
        # GradNorm-style methods re-set these from gradient norms, once per
        # epoch in the driving paper  [arxiv:2606.02979 alg. 1]
        total += weights[name] * loss
    # CAGrad keeps the per-task gradients but merges them into one update
    # direction, with no fixed weights  [arxiv:2110.14048 alg. 1]
    # Florence-2 replaces this loop with one token loss and no weights
    # [arxiv:2311.06242 eq. 1]. MAGVIT and Unified-IO also use one token
    # cross-entropy over a shared vocabulary; MAGVIT masks condition positions
    # [arxiv:2212.05199 eq. 3] [arxiv:2206.08916 §3.3]
    # T0 also has one likelihood over prompted targets, with no weights; it
    # samples its mixture with a per-dataset cap  [arxiv:2110.08207 §5]
    # Med-PaLM M draws each batch with mixture ratios instead; its loss form
    # is not stated  [arxiv:2307.14334 §4.2]
    # Per-annotator variant: the loop runs over annotators, and each head's
    # loss covers only the instances that annotator labelled  [arxiv:2110.05719 §3.4]
    # The fixation and camouflage terms are summed with lambda = 1
    # [arxiv:2103.04011 eq. 3]
    # TCP sums its branch losses with fixed weights, 1 for each branch
    # [arxiv:2206.08129 eq. 6]
    # Painter has no per-task weights: one smooth L1 over masked output pixels
    # [arxiv:2212.02499 §3.2]
    # LIQE sets its weights from loss descent ratios, softmax over tasks
    # [arxiv:2303.14968 eq. 11]
    return total

Contradictions

• Rank correlation in the Omnidata MTL comparison. The text and the table caption say the within-task rank correlation is indistinguishable from zero 2110.04994 §5.3 2110.04994 tab. 6. The table's own summary cells give 0.43 and 0.2 2110.04994 tab. 6. Four methods per column is a small sample for a rank correlation. Both sides are reported here; the page does not resolve them.
• Collision claims in the motion planner. The text claims a clearly lower future collision rate at every horizon 2101.06679 §4.1. The table gives 0.00 for every method at 0.5 s and near-equal values at 1.0 s 2101.06679 tab. 1.
• Joint-training gains on different tasks. MAGVIT reports large gains from joint training on video 2212.05199 §4.4. Omnidata reports that method rankings change with the dataset 2110.04994 tab. 6, and the motion planner's joint model is mixed against planning alone 2101.06679 tab. 3. The camouflage paper's ablation shows a joint gain on its COD metrics and a small loss on ranking 2103.04011 tab. 5. TCP's shared-backbone model beats its two-model ensemble on the driving score, but the ensemble is twice as large 2206.08129 tab. 4. Painter's joint model beats separate models on most dense tasks, with two small exceptions 2212.02499 tab. 2. LIQE's extra tasks help mean SRCC by 0.007 and cost accuracy on the auxiliary tasks 2303.14968 tab. 5. These are different tasks and metrics, so the page does not treat them as a direct conflict.
• Loss weights for one design. The report generator fixes its four terms at weight 1.0 1711.08195 §4.2. The rumour paper states a plain sum over task losses 1806.03713 §4.2, but its released code uses 0.3, 0.4 and 0.3 see the card. The sentiment paper gives equal priority and no formula 1901.08014 §2. None of these cards reports a sweep over the weights, so the page does not say which choice is better.
• Joint training in the driving networks. YOLOP's joint model is mixed against its single-task models: level on detection, lower on lane lines 2108.11250 tab. 5. Its region-based R-CNNP baseline drops on both detection and segmentation when trained jointly 2108.11250 tab. 6. The paper calls the time saving substantial 2108.11250 §4.3.2. The compact network above beats its combined baselines on three of four sets 2606.02979 tab. 2. These are different networks, tasks and baselines, so the page does not treat them as a direct conflict.
• CityScapes radius for CAGrad. The text gives c = 0.2 2110.14048 §5.2. The appendix and the reported table row give c = 0.4 2110.14048 app. B.2 2110.14048 tab. 2. The page does not resolve this; see the card for the code, which runs 0.4 and 0.6.

Papers

• On the Automatic Generation of Medical Imaging Reports: a chest x-ray report generator trained on a summed tag, stop, word and attention-coverage objective, with a tag-classification task that feeds the decoder's attention.
• End-to-end Interpretable Neural Motion Planner: a shared CNN for detection, forecasting and a cost volume, trained with a perception loss plus a weighted max-margin planning loss.
• Omnidata: a dataset pipeline whose multi-task comparison pits single-task training against a shared-encoder baseline, MTAN and Cross-stitch.
• Unified-IO: A unified model for vision, language, and multi-modal tasks: one T5-style encoder-decoder over a shared token vocabulary, trained on 95 datasets with no task-specific heads.
• VIMA: General Robot Manipulation with Multimodal Prompts: one prompted robot policy trained by behaviour cloning across 17 task templates, with cross-attention to a text-and-image prompt.
• MAGVIT: Masked Generative Video Transformer: one masked token transformer trained with one loss over ten video tasks, where each task is set by its condition pixels.
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning: an ablation study of a template and task mixture for T5-XL, covering zero- and few-shot templates, input inversion and mixture balancing.
• Towards Generalist Biomedical AI: a PaLM-E-based model fine-tuned on 14 biomedical tasks as text-answer instructions, with mixture sampling and no task heads; a single-run ablation drops one classification task.
• Multitask Prompted Training Enables Zero-Shot Task Generalization: T5 fine-tuned on 62 prompted datasets, then tested zero-shot on held-out NLI, coreference, sentence-completion and WSD tasks.
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks: one prompt-driven model trained with one token-level loss over captioning, detection, grounding, segmentation and OCR.
• A Taxonomy of Deep Convolutional Neural Nets for Computer Vision: a 2016 survey of CNN designs with no experiments of its own; its multi-task section covers a multi-task loss with auxiliary tasks and task-specific sub-networks 1601.06615 §3.5.
• All-in-one: Multi-task Learning for Rumour Verification: an LSTM with stance, detection and veracity heads that share recurrent layers, trained on Twitter reply branches.
• Neural Additive Models: Interpretable Machine Learning with Neural Nets: per-feature neural nets whose multitask variant mixes shared subnets with a learned weighted sum per task.
• Sentiment and Sarcasm Classification with Multitask Learning: a shared GRU with attention pooling per task, and a neural tensor network that fuses the two task vectors.
• Multi-task Self-Supervised Learning for Human Activity Detection: a 1-D CNN with eight binary heads that predict which signal transform was applied to an accelerometer window, then reused for activity recognition.
• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System: one T5 model pre-trained on eleven partially annotated dialogue corpora, with each sub-task written as a text prompt.
• Multi-Game Decision Transformers: one transformer over offline Atari trajectories from 41 games, trained with next-token prediction over return, action and reward tokens.
• Anabranch Network for Camouflaged Object Segmentation: a saliency FCN with a camouflage classifier whose probability scales the pixel map, trained in two steps, with a summed-loss joint variant.
• Conflict-Averse Gradient Descent for Multi-task Learning: a gradient-merging update that starts from the average task gradient and stays within a radius of it, with a sampled variant for many tasks.
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations: one BERT encoder with a head per annotator, trained on each annotator's labels, with the majority vote taken over the heads.
• Simultaneously Localize, Segment and Rank the Camouflaged Objects: a ResNet-50 with a fixation decoder that guides a camouflage decoder through reverse attention, trained with two summed losses; the ranking branch is not released.
• Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion: one network with per-input encoders and per-output decoders for four driving perception outputs, trained with GradNorm-style loss weights updated once per epoch.
• YOLOP: You Only Look Once for Panoptic Driving Perception: one shared encoder feeding a vehicle detector and two segmentation heads for drivable area and lane lines, trained end to end on BDD100K.
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: one camera encoder with a trajectory branch and a multi-step control branch, blended at test time, trained by imitating an RL expert.
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning: one masked-image model trained on stitched same-task pairs over seven dense output types, with in-context prompts instead of task heads.
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective: a fine-tuned CLIP that scores images against 495 joint quality, scene and distortion labels, with dynamic loss weights.