Hierarchical classification

Overview

Many label sets form a tree: a general class has finer children, and each finer class has one general parent. A flat classifier ignores that structure and treats a fine label and its parent as unrelated. The variants here predict the whole chain of labels and tie the scores together.

The source assumes a tree with one parent per class. The root is the most general class, and every class counts as its own ancestor and descendant 2203.14335 §3.1. The paper does not say how to handle a class with two parents.

The invariant all variants aim for: if a node is positive, all of its ancestors are positive, and if a node is negative, all of its descendants are negative 2203.14335 §3.2.1. The variants differ in how they enforce this, and in whether they add a metric term on top.

Variants

Flat softmax over the leaves (baseline)

One categorical cross-entropy over the finest classes. The tree plays no part in the output or the loss 2203.14335 eq. 1. In the paper's dense prediction setting, the baseline reaches 81.86 / 68.17 / 37.43 mIoU at the coarse, middle and fine levels on Mapillary Vistas 2.0 [arxiv:2203.14335 tab. 5].

Per-node sigmoid head with plain binary cross-entropy

Each node of the tree, internal nodes included, gets its own sigmoid score, trained with binary cross-entropy over all nodes 2203.14335 eq. 2. The scores do not sum to one. At inference the method searches the valid chains that run from a leaf up to the root, and keeps the one whose node scores add up highest 2203.14335 eq. 3. On its own this head does not help. It scores 81.56 / 67.61 / 37.26 on the same setting, slightly below the softmax baseline 2203.14335 tab. 6.

Tree-min probabilities with focal weighting

The node probability is taken from the tree rather than from the raw score. A positive node uses the minimum score over its ancestors, and a negative node uses the maximum over its descendants 2203.14335 eq. 4. Binary cross-entropy on these values satisfies both constraints by construction, so a violating prediction is penalised 2203.14335 §3.2.1, eq. 5. A focal factor with exponent 2 is added on top 2203.14335 eq. 6, tab. 7. The focal version scores 84.17 / 69.62 / 39.17 on the same setting, against 83.48 / 69.13 / 38.69 for the version without the focal factor [arxiv:2203.14335 tab. 6]. This variant does not need the projection head, which the metric term alone uses 2203.14335 §3.3.

Tree-triplet metric term

A triplet loss is added on a 256-dimensional projection of the pixel embeddings. Triplets are chosen so that the tree distance from anchor to positive is smaller than from anchor to negative. The embedding distance is half of one minus cosine similarity, and the margin depends on the tree distances 2203.14335 §3.2.2, §3.3, eq. 7, eq. 8. The projection is dropped once training ends, so inference cost is unchanged 2203.14335 §3.3. Added to the focal tree-min loss, the combined model scores 85.27 / 71.40 / 40.16 2203.14335 tab. 5. A margin that follows the hierarchy beats a constant margin, 85.27 against 84.66 at the coarse level [arxiv:2203.14335 tab. 8]. Cosine distance beats Euclidean distance, 85.27 against 84.23 at the coarse level 2203.14335 tab. 9.

When to use which

The reported gains over DeepLabV3+ on Mapillary are 3.41 at the coarse level, 3.23 at the middle and 2.73 at the fine level 2203.14335 tab. 1. The gains therefore shrink towards the finest labels. The same method also reports gains on three further datasets and on three backbones 2203.14335 tab. 1, tab. 2, tab. 3, tab. 4. All numbers are author-reported.

Implementation sketch

Training step, using the paper's equations. Items marked "optional" follow the paper's ablations and are not needed for the focal tree-min variant.

# Hierarchical head and loss, one training step  [§3.1] [§3.2] [§3.3]
# Input:  image batch, leaf labels, tree with parent links
# Output: scalar loss

def hierarchical_loss(images, leaf_labels, step):
    # Score every tree node with a sigmoid, internal nodes included  [§3.1]
    feats = encoder(images)
    scores = sigmoid(node_head(feats))            # (B, H, W, num_nodes)
    # Binary target per node: 1 if the leaf lies under that node  [§3.2.1]
    node_targets = ancestor_targets(leaf_labels, tree)
    # Tree-min probability: minimum over ancestors if the node is positive,
    # maximum over descendants if negative  [eq. 4]
    probs = where(node_targets,
                  min_over_ancestors(scores),
                  max_over_descendants(scores))
    # Focal binary cross-entropy with exponent 2  [eq. 6] [tab. 7]
    loss = focal_bce(probs, node_targets, gamma=2.0)
    # Optional: tree-triplet term on a 256-d projection  [§3.2.2] [§3.3]
    if use_triplet:
        emb = proj_head(feats)                    # (B, H, W, 256)
        # Weight rises on a cosine curve from 0 to 0.5  [§3.3]
        beta = 0.25 * (1 - cos(pi * step / total_steps))
        loss += beta * triplet_loss(emb, leaf_labels, tree)
    return loss

Inference picks the best root-to-leaf path. For a two-level tree, this is the leaf whose own score plus its parent's score is largest [arxiv:2203.14335 eq. 3]:

# Inference: best feasible root-to-leaf path  [§3.1] [eq. 3]
def predict(scores):
    # Sum node scores along the path from each leaf up to the root
    path_score = sum_along_path_to_root(scores)   # one value per leaf
    # Pick the leaf with the largest path score
    return argmax(path_score)

Contradictions

None between papers, because the page has one source.

Within the one source, the text and the tables differ on small points. The text gives gains over DeepLabV3+ of 2.69, 3.21 and 3.40 points [arxiv:2203.14335 §4.2]. Table 1 gives 2.73, 3.23 and 3.41 2203.14335 tab. 1. This page uses the table values. Section 3.2.1 says a gamma of 0 makes the focal loss equal the tree-min loss. Table 7's gamma-0 row (84.47 at the coarse level) does not match the tree-min row of Table 6 (83.48) [arxiv:2203.14335 §3.2.1, tab. 6, tab. 7].

Pitfalls

• Released code is partial. The paper's ablation favours the focal tree-min loss 2203.14335 tab. 6. The released Cityscapes loss calls a non-focal version, although a focal version exists in the same file hiera_triplet_loss_cityscape.py. The PASCAL and LIP path does use the focal loss [code: Pytorch/mmseg/models/losses/rmi_hiera_triplet_loss.py#L161]. The release has no Mapillary, OCRNet or MaskFormer configs, so the Transformer results in the paper cannot be reproduced from it. Check which loss a given config trains before comparing numbers.
• Margin units. The paper's distance is half of one minus cosine. A constant margin of 0.6 on one minus cosine is not the same as 0.6 on the paper's scale 2203.14335 §3.2.2, eq. 7.
• Triplet sampling. The paper samples triplets from the batch but does not say how. Sampling choices change which pixels supply the triplets 2203.14335 §3.2.2.
• Loss normalisation. The paper does not say whether the loss is summed or averaged over pixels and classes. This sets the relative weight of the metric term 2203.14335 eq. 2, eq. 6.
• Hardware. The paper's timing is for its own hardware. The per-batch overhead depends on the setup 2203.14335 tab. D.1.

Papers

• Deep Hierarchical Semantic Segmentation: proposes a per-node sigmoid head with a focal tree-min loss and a tree-triplet metric term, evaluated on four segmentation datasets.