Multi-label classification

Overview

A multi-label input carries several correct labels at once, so the target is a binary vector rather than one class. The usual setup gives each label its own logit and sigmoid, trained with binary cross-entropy. Each label is scored on its own, so the model cannot use the fact that some labels co-occur and others rarely appear together 2011.14027 §1.

The cards in this wiki change that default in different places. Some models add label dependencies, some handle targets that are uncertain, some down-weight candidate labels that may be wrong, and one keeps the sigmoid classifier as the first stage of weakly supervised segmentation, because a softmax over classes is not valid when an image has several true classes at once 2203.00962 §1 2203.00962 §5.2. Two more cards change the loss or the head: one uses an asymmetric loss on the sigmoid outputs, and one splits the classes by WordNet depth and puts a softmax on each depth 2104.10972 §3.2 2104.10972 §3.3. A last route writes the label rules as first-order logic and trains the sigmoid outputs so that those rules hold. Logic Tensor Networks need explicit exclusion rules for this, because a sigmoid head does not stop two labels from both being on 2012.13635 §4.3.

"Partial" means two different things in these cards. In C-Tran, some target labels are known at test time. In PML-CD, the training labels are candidates that may include wrong positives. The two settings are separate and are named separately below.

Variants

Independent per-label sigmoid head (BCE)

Shared image features feed one linear output per label. The loss is binary cross-entropy over all labels, and each label's sigmoid is read on its own 2203.00962 §4.1 eq. 1, eq. 4. ReCAM's first stage is this model, with global average pooling in front of the head 2203.00962 §3. C-Tran reports a plain ResNet-101 row for reference: 77.3 mAP on COCO-80 and 30.9 on VG-500 2011.14027 tab. 1 2011.14027 tab. 2. The card does not describe that baseline's head beyond the row, so use it as a reference point only. Card: General Multi-label Image Classification with Transformers.

MovieNet uses binary cross-entropy for its 21-genre task, scored with a 0.5 threshold for its precision and recall figures 2007.10937 §4.1 2007.10937 app. C.1. The author-reported mAP is 46.88 for a ResNet-50 on 1.6M photos and 45.23 for a TRN-R50 on trailers, against 43.70 for TSN-R50 2007.10937 tab. 3. The inputs differ between rows, so these figures do not compare losses. The same card also tags actions and places per clip as a multi-label task. The best action result is SlowFast at 23.52 mAP, and the best place result is TSN at 8.33 mAP 2007.10937 §4.4 2007.10937 tab. 8. Card: MovieNet: A Holistic Dataset for Movie Understanding.

Asymmetric loss on the sigmoid head (ASL)

The head is still one sigmoid per class. The change is in the binary loss, which the paper calls asymmetric. Its multi-label pretraining uses a focusing parameter of 4 on negatives and 0 on positives 2104.10972 §3.2 2104.10972 app. B. The appendix says several negative-focusing values were tried to maximise mAP 2104.10972 app. B.

On the pretraining data, ASL reaches 48.5 micro mAP and 74.7 macro mAP. Cross- entropy reaches 47.3 and 73.9, and focal loss 47.4 and 74.1 2104.10972 tab. 6. With multi-label pretraining, ASL beats cross-entropy on all seven downstream datasets, by 0.3 to 1.6 points 2104.10972 tab. 7. The gains are the paper's own, and the caveats below apply. Card: ImageNet-21K Pretraining for the Masses.

One softmax per hierarchy depth (semantic softmax)

Use this when each image carries labels at several levels of a class hierarchy, such as the WordNet ancestors of its class. The classes are grouped by depth, and each group gets its own softmax. An image labelled at depth d also trains the softmaxes at depths 0 to d, which see its ancestor labels 2104.10972 §2.2 2104.10972 §3.3. The per-depth losses are summed with weights 2104.10972 eq. 3. The paper uses 11 depths on ImageNet-21K-P 2104.10972 §2.2.

Compared with the single-label softmax, semantic softmax is higher on all seven transfer datasets. Compared with sigmoid multi-label pretraining, it is higher on six; the exception is CIFAR-100 2104.10972 tab. 2. Examples: ImageNet-1K top-1 is 81.4 against 81.1 and 81.0, MS-COCO mAP is 81.3 against 80.8 and 80.6, and Pascal-VOC mAP is 89.7 against 88.1 and 87.9 2104.10972 tab. 2. These are transfer results after pretraining. Adding a semantic distillation term raises every column further, but that term is not in the release (see Caveats) 2104.10972 tab. 2 2104.10972 §3.4. Card: ImageNet-21K Pretraining for the Masses.

Label-graph methods (named baselines only)

Earlier label-dependency methods that C-Tran compares against, ML-GCN, SSGRL and KGGR, rely on label graphs built from co-occurrence statistics 2011.14027 §5. C-Tran's comparison table lists them at 83.0, 83.8 and 84.3 mAP on COCO-80 2011.14027 tab. 1. C-Tran also states that such methods cannot take partial evidence at test time 2011.14027 §5. No card on wikillm covers these methods, so the numbers above are only what C-Tran reports for them.

Label tokens in a Transformer with label-mask training

C-Tran gives each label a learned token and a state (unknown, negative or positive). The label tokens and image grid tokens go through a Transformer encoder together. Each label's output token then gets its own linear readout 2011.14027 §3.1 2011.14027 §3.2 2011.14027 §3.3. During training, a random share of labels is hidden and the loss is taken only on the hidden labels 2011.14027 §3.4. Without the image, mAP drops to 3.60 on COCO-80 at 0% known labels, so the image pathway carries most of the signal when no labels are known 2011.14027 tab. 5.

Reported results, mAP on the regular test split: 85.1 on COCO-80, 38.4 on VG-500 and 34.3 on COCO-1000 2011.14027 tab. 1 2011.14027 tab. 2 2011.14027 tab. 5. Removing label-mask training costs 0.3 points on COCO-80 and 0.1 on VG-500 at 0% known labels, but 1.6 points on VG-500 and 18.4 points on NEWS-500 at 50% known labels 2011.14027 tab. 5. These are single runs with no variance reported 2011.14027 §4.

Using known labels at inference (partial and extra labels)

This is a setting rather than a separate model. Some target labels are known at test time (partial labels), or related labels outside the target set are known (extra labels). In C-Tran, known labels are given their state in the input and the model returns predictions in one pass, with no iterative refinement 2011.14027 §4.2 2011.14027 §4.3.

Against Feedbackprop, the paper's only partial-label baseline, C-Tran's mAP on unknown labels rises with the known share. On NEWS-500 it goes from 18.1 at 0% to 39.4 at 75%, against 14.7 and 25.9 for Feedbackprop 2011.14027 tab. 3. On COCO-80 the same comparison is 85.1 to 86.0 against 80.1 to 80.9 2011.14027 tab. 3. The extra-label experiment uses CUB-312, a 200-class task with extra label groups, so it is not a multi-label result 2011.14027 §4.3. The paper assumes known labels are correct and does not test noisy ones 2011.14027 §2 2011.14027 §3.4.

Uncertain targets: one state per label (CheXpert)

Each target here is yes, no or uncertain, for 14 observations per chest radiograph. The labels were extracted from radiology reports by a rule-based labeler, so "uncertain" marks hedged report wording rather than an annotator's doubt 1901.07031 § Method. The paper compares five ways to train a DenseNet121 on these targets 1901.07031 § Uncertainty Approaches:

• U-Ignore: binary cross-entropy summed over observations, skipping each uncertain entry.
• U-Zeros and U-Ones: map every uncertain entry to 0 or to 1, then apply ordinary binary cross-entropy.
• U-SelfTrained: train U-Ignore to convergence, then use its predicted probability as the target for each uncertain entry.
• U-MultiClass: a three-way softmax (negative, positive, uncertain) per observation. At test time the positive probability is taken after renormalising over negative and positive.

No approach wins on every observation. On the 200-study validation set, U-Ones has the highest AUROC for Atelectasis (0.858) and U-MultiClass for Cardiomegaly (0.854). Within each observation, the five approaches differ by 0.01 to 0.04 AUROC 1901.07031 tab. 3. The paper reports U-Ones beating U-Zeros on Atelectasis (p = 0.03) and U-MultiClass beating U-Ignore on Cardiomegaly (p < 0.01), with no significant difference for Consolidation, Edema or Pleural Effusion 1901.07031 § Results. On the test set, each observation uses the approach that scored best on validation: U-Ones for Atelectasis and Edema, U-MultiClass for Cardiomegaly and Pleural Effusion, and U-SelfTrained for Consolidation 1901.07031 § Test Results. Card: CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.

Noisy candidate labels: calibrated per-entry weights (PML-CD)

Here each image carries candidate labels, and some candidate positives are wrong. PML-CD keeps the BCE loss but weights each candidate positive by an estimate of the chance that it is correct, given the model's confidence in it. The paper argues that this optimal weight equals the calibrated probability of the candidate being true 2610.17635 §3 2610.17635 eq. 3–4. Negatives keep weight 1 2610.17635 alg. 1.

The weight comes from a small pre-trained calibrator. It maps a 20-bin histogram of the confidences on candidate positives to a reliability per bin, and each confidence gets a weight by interpolating between neighbouring bins 2610.17635 §4.1 2610.17635 eq. 7. The calibrator was trained on VOC 2007 at noise rates from 10% to 90%, then reused unchanged on other datasets and noise rates 2610.17635 §5.1 2610.17635 §5.2. A second phase adds a prototype-cosine loss on class embeddings 2610.17635 §4.2 2610.17635 eq. 13.

Author-reported mAP, at 20% and 80% noise. On VOC, PML-CD gets 88.83 and 71.70, against 87.73 and 48.86 for BCE on the raw candidates and 64.46 for ML-Decoder at 80% 2610.17635 tab. 1. On COCO at 80% noise, PML-CD gets 62.10 against 50.47 for BCE 2610.17635 tab. 1. The paper reports PML-CD as best in 15 of 16 settings; the exception is VOC at 20% noise, where LL-R gets 88.99 2610.17635 §5.2. In the ablation, the second-phase prototype loss lifts VOC at 80% noise from 66.91 to 71.70 2610.17635 tab. 3. The noise in every reported setting is synthetic: each 0 is flipped to 1 with probability ρ 2610.17635 §5.1. Card: Calibrated Disambiguation for Partial Multi-label Learning.

Logical rules as the training objective (LTN)

Logic Tensor Networks treat each formula as a fuzzy truth value in [0, 1], and training maximises the satisfiability of a knowledge base built from those formulas 2012.13635 §2.2 2012.13635 §3.2. For multi-label data, the paper uses one sigmoid predicate P(x, l) over four labels: blue, orange, male and female. It adds four axioms that mark the given positives, plus two exclusion rules, one for each colour pair and one for each sex pair. No negative examples are given 2012.13635 §4.3. The experiment uses a 200-example crabs dataset with 5 features 2012.13635 §4.3.

The paper reports the multi-label case as a trend, not as an accuracy comparison. The rule "blue implies not orange" rises towards true during training, while "blue implies orange" and "blue implies male" fall towards false 2012.13635 §4.3 2012.13635 fig. 12. The card's results give no multi-label accuracy figure, although the paper defines accuracy as one minus the Hamming loss at threshold 0.5 2012.13635 §4.3. Card: Logic Tensor Networks.

BCE plus softmax on isolated class features (localisation)

Classification is not the goal here. The goal is better class activation maps for pixel masks. ReCAM keeps BCE on the classifier and adds a second softmax term. Each class's feature map is first masked by that class's activation map, so the second term sees one class per feature 2203.00962 §4.1 eq. 5–8. Softmax alone on the full multi-label target is worse: on VOC seed masks it gives 44.6 mIoU against 48.8 for BCE alone, and 27.9 against 33.1 on MS COCO 2203.00962 tab. 1. The best variant reaches 54.8 mIoU on VOC seeds 2203.00962 tab. 2. The paper reports only segmentation mIoU, so classification mAP is not measured. Card: Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation.

When to use which

Implementation sketch

Common core: one logit per label, binary cross-entropy, sigmoid at inference. The label-token variant replaces the head with the Transformer described above.

# Common core for multi-label training
# [arxiv:2203.00962 §4.1] [arxiv:2011.14027 §3.4]
# Input:  image (3, H, W); targets y (L,) with 0 or 1
# Output: scalar loss

def multilabel_loss(image, y, backbone, head):
    # Shared image features, pooled to one vector  [arxiv:2203.00962 §4.1]
    feat = backbone(image)                       # (d,)
    # One logit per label; no softmax across labels  [arxiv:2203.00962 §1]
    logits = head(feat)                          # (L,)
    # Binary cross-entropy per label  [arxiv:2011.14027 §3.4]
    # The paper does not state how the terms are reduced
    return bce_with_logits(logits, y)

# Inference  [arxiv:2011.14027 §4.1]
def predict(image, backbone, head):
    # Sigmoid per label, then a fixed threshold
    probs = sigmoid(head(backbone(image)))       # (L,)
    return probs >= 0.5                          # threshold 0.5 as in C-Tran

Notes:

• Label-token variant: replace head(feat) with a Transformer over image grid tokens and label tokens, where each label token carries a state embedding (unknown, negative or positive). Mask random labels during training and take the loss on the hidden ones only 2011.14027 §3.1 2011.14027 §3.4.
• Known labels at test time: set their states in the input and keep the rest unknown, then run one forward pass 2011.14027 §4.2.
• Localisation variant: the softmax term is computed on per-class masked features, and the BCE term stays on the classifier 2203.00962 §4.1.
• Uncertain targets: before the loss, mask the entry (U-Ignore), set it to 0 or 1 (U-Zeros, U-Ones), replace it with the teacher's probability (U-SelfTrained), or switch the observation to a three-way softmax (U-MultiClass) 1901.07031 § Uncertainty Approaches.
• Candidate weights: multiply the BCE term of each candidate positive by its calibrated weight, and keep negatives at weight 1 2610.17635 alg. 1 2610.17635 eq. 1. The official code applies a further rescaling of the weights; see Caveats.
• Asymmetric loss: replace the bce_with_logits term with the asymmetric loss, using a focusing parameter of 4 on negatives and 0 on positives 2104.10972 app. B. This sketch does not restate the formula; use the paper's definition 2104.10972 §3.2.
• Semantic softmax: replace the head with one softmax per depth, and train each image at depths 0 to the depth of its own label 2104.10972 §3.3. Take the depth weights from the released code, not from eq. 2 (see Caveats).
• Logical rules: keep one sigmoid output per label, then replace the BCE term with a loss of one minus the satisfiability of the rules. Each rule is a fuzzy formula, using the stable product operators and a p-mean for the universal quantifier 2012.13635 §2.4 2012.13635 §3.2. Each exclusion rule is a universal over the instances, and the rule's truth is that of the negated conjunction of the two labels 2012.13635 §4.3.

Caveats

• C-Tran and ReCAM report single runs with no variance 2011.14027 §4 2203.00962 §5.1. PML-CD does not state its run count 2610.17635 §5.1. CheXpert's AUROC intervals are DeLong intervals on the validation set, which reflect sampling error in that set, not variation across training runs 1901.07031 tab. 3. Differences of a few tenths of a point are not established as real.
• The semantic-softmax paper gives 3 runs for downstream datasets under 150,000 images 2104.10972 app. D. The card states no run count for its pretraining tables 2104.10972 tab. 2 2104.10972 tab. 6 2104.10972 tab. 7.
• The ASL comparison may not be like for like. The paper tried several negative-focusing values to maximise mAP 2104.10972 app. B, and the card does not say whether the cross-entropy baseline got the same tuning.
• The multi-label pretraining baseline is not fully specified. The card does not say whether ancestor labels are positives in the sigmoid baseline, so semantic softmax and sigmoid pretraining may not use the same targets 2104.10972 §3.2.
• Semantic KD is not usable as described. The teacher is not named, the weight between KD and classification losses is not stated, and the rule for teacher confidence is ambiguous 2104.10972 §3.4 2104.10972 app. F. The released code has the KD helper but no training script that calls it (see the card). Treat the KD column of tab. 2 as author-reported only 2104.10972 tab. 2.
• The depth weights in eq. 2 and eq. 3 do not match the released code, and eq. 2 is undefined at depth 0 as printed. Pick one version and state it 2104.10972 eq. 2 2104.10972 eq. 3.
• ImageNet-21K-P is derived from ImageNet-21K. The paper states no separate dataset licence, and the release says its use is subject to image-net.org terms. Check those terms before you redistribute derived data 2104.10972 §2.1.
• C-Tran's released training masks are off for VG-500 and NEWS-500 in the shipped code, while the paper describes label-mask training for all datasets 2011.14027 §3.4. The paper does not explain this, so the VG-500 and NEWS-500 rows of its tables may not match the method as described.
• Attention in C-Tran runs over 324 image tokens plus one token per label 2011.14027 §3.2. Its experiments use up to 1000 labels 2011.14027 tab. 5; the roughly 2,000-label limit is an estimate, not a tested maximum 2011.14027 §4.4.
• CheXpert's labels come from a rule-based labeler, not from annotators, and its uncertainty comparison uses one hospital's data 1901.07031 § Dataset. The radiologist comparison has no statistical test 1901.07031 § Limitations. The paper's CC BY 4.0 licence covers its text and figures, not the dataset 1901.07031 § Pitfalls.
• PML-CD's official code rescales the weights after the calibrator, uses a hard threshold of 0.7 where the paper says 0.6, and runs early stopping on a loader built from the test split 2610.17635 § Paper vs code. The paper's "optimal" weighting uses true labels, so it is an upper bound, not a method you can deploy 2610.17635 tab. 2.
• LTN's multi-label evidence is one qualitative trend on a small crabs dataset, with no accuracy comparison against a baseline 2012.13635 §4.3. The paper's only numeric baseline comparison is MNIST addition, which is not a multi-label task 2012.13635 tab. 2.
• LTN's stable operators need a small positive constant the paper does not state 2012.13635 §2.4. Set it yourself and record the value.
• PML-CD's two tables disagree on the VOC prototype ablation: 71.70 at 80% matches, but the 20% to 60% values differ between the ablation and the main table 2610.17635 tab. 1 2610.17635 tab. 3.
• MovieNet's paper licence (CC BY 4.0) covers its text, not the films, trailers, photos or labels. The project page describes access under a user agreement, and the card records a 60K vs 68K trailer-count conflict. Check the data terms before any commercial use 2007.10937 § Pitfalls.

Contradictions

Most results do not conflict across the cards, because they measure different things: classification mAP for C-Tran, PML-CD and CheXpert, top-1 and mAP after pretraining for the semantic-softmax paper, and seed-mask mIoU for ReCAM. Three points still sit uneasily together.

ReCAM finds a softmax over the full multi-label target worse than BCE, 44.6 against 48.8 mIoU on VOC seeds 2203.00962 tab. 1. The semantic-softmax paper finds one softmax per hierarchy depth better than sigmoid pretraining on six of seven transfer datasets 2104.10972 tab. 2. These differ in target (the full multi-label set against ancestor labels), in stage (segmentation seeds against pretraining), and in measure, so neither refutes the other.

CheXpert's U-MultiClass, which uses a three-way softmax per observation, beats U-Ignore on Cardiomegaly 1901.07031 § Results. That is a different setting again, and it does not settle the ReCAM point either.

The LTN card adds no number that conflicts with these, because its only multi-label result is a qualitative trend 2012.13635 §4.3.

Papers

• General Multi-label Image Classification with Transformers: label tokens with a state embedding in a Transformer, label-mask training, and inference with partial and extra labels.
• Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation: keeps BCE on a multi-label classifier and adds a softmax term on isolated class features to improve pseudo masks.
• CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison: five ways to train on uncertain labels, compared on validation AUROC for five observations.
• Calibrated Disambiguation for Partial Multi-label Learning: calibrated per-entry weights for candidate positives, plus a prototype-cosine loss, tested under synthetic label noise.
• ImageNet-21K Pretraining for the Masses: semantic softmax over WordNet depths, and asymmetric loss for multi-label pretraining, with transfer results on seven datasets.
• Logic Tensor Networks: first-order rules as a differentiable training objective, with a multi-label crabs experiment that uses exclusion axioms and no negatives.
• MovieNet: A Holistic Dataset for Movie Understanding: a film dataset with a 21-genre multi-label task trained with BCE, plus multi-label action and place tagging per clip; results are author-reported.