---
kind: concept
slug: hierarchical-classification
title: Hierarchical classification
summary: >-
  Predicting labels that form a class tree, with one score per node instead of one softmax over the leaves, and losses that keep ancestor and descendant scores consistent. The only source so far is a segmentation paper, which reports author-measured gains on four datasets. The trade-off is extra loss terms and, for the metric term, about 27% more training time per batch on one dataset.
tags: [hierarchy, segmentation, loss-function]
papers: ["2203.14335"]
related: [semantic-segmentation, multi-label-classification, metric-learning, focal-loss]
status: draft
updated_at: 2026-11-09
---

## Overview

Many label sets form a tree: a general class has finer children, and each
finer class has one general parent. A flat classifier ignores that structure
and treats a fine label and its parent as unrelated. The variants here predict
the whole chain of labels and tie the scores together.

The source assumes a tree with one parent per class. The root is the most
general class, and every class counts as its own ancestor and descendant
[arxiv:2203.14335 §3.1]. The paper does not say how to handle a class with two
parents.

The invariant all variants aim for: if a node is positive, all of its
ancestors are positive, and if a node is negative, all of its descendants are
negative [arxiv:2203.14335 §3.2.1]. The variants differ in how they enforce
this, and in whether they add a metric term on top.

## Variants

### Flat softmax over the leaves (baseline)

One categorical cross-entropy over the finest classes. The tree plays no part
in the output or the loss [arxiv:2203.14335 eq. 1]. In the paper's dense
prediction setting, the baseline reaches 81.86 / 68.17 / 37.43 mIoU at the
coarse, middle and fine levels on Mapillary Vistas 2.0 [arxiv:2203.14335
tab. 5].

### Per-node sigmoid head with plain binary cross-entropy

Each node of the tree, internal nodes included, gets its own sigmoid score,
trained with binary cross-entropy over all nodes [arxiv:2203.14335 eq. 2]. The
scores do not sum to one. At inference the method searches the valid chains that run from a leaf up
to the root, and keeps the one whose node scores add up highest
[arxiv:2203.14335 eq. 3].
On its own this head does not help. It scores 81.56 / 67.61 / 37.26 on the same
setting, slightly below the softmax baseline [arxiv:2203.14335 tab. 6].

### Tree-min probabilities with focal weighting

The node probability is taken from the tree rather than from the raw score. A
positive node uses the minimum score over its ancestors, and a negative node
uses the maximum over its descendants [arxiv:2203.14335 eq. 4]. Binary
cross-entropy on these values satisfies both constraints by construction, so a
violating prediction is penalised [arxiv:2203.14335 §3.2.1, eq. 5]. A focal
factor with exponent 2 is added on top [arxiv:2203.14335 eq. 6, tab. 7]. The
focal version scores 84.17 / 69.62 / 39.17 on the same setting, against 83.48 /
69.13 / 38.69 for the version without the focal factor [arxiv:2203.14335 tab.
6]. This variant does not need the projection head, which the metric term
alone uses [arxiv:2203.14335 §3.3].

### Tree-triplet metric term

A triplet loss is added on a 256-dimensional projection of the pixel
embeddings. Triplets are chosen so that the tree distance from anchor to
positive is smaller than from anchor to negative. The embedding distance is
half of one minus cosine similarity, and the margin depends on the tree
distances [arxiv:2203.14335 §3.2.2, §3.3, eq. 7, eq. 8]. The projection is dropped
once training ends, so inference cost is unchanged [arxiv:2203.14335 §3.3]. Added to the focal
tree-min loss, the combined model scores 85.27 / 71.40 / 40.16
[arxiv:2203.14335 tab. 5]. A margin that follows the hierarchy beats a
constant margin, 85.27 against 84.66 at the coarse level [arxiv:2203.14335
tab. 8]. Cosine distance beats Euclidean distance, 85.27 against 84.23 at the
coarse level [arxiv:2203.14335 tab. 9].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Labels are flat, with no class tree | Not this family | The constraints are defined on tree nodes [arxiv:2203.14335 §3.1] |
| Tree known, only the output head may change, no extra embedding | Per-node sigmoid head with focal tree-min loss | Without the metric term it gives 84.17 against 81.86 at the coarse level [arxiv:2203.14335 tab. 5] |
| Tree known, GPU training budget allows extra cost | Add the tree-triplet term | Adds about 1.1 points at the coarse level over the focal tree-min loss alone [arxiv:2203.14335 tab. 5]. Time per batch rises from 0.93 to 1.18 s on Mapillary (the paper's own timing) [arxiv:2203.14335 tab. D.1] |
| Plain sigmoid per node, no tree-aware loss | Not supported by this source | It scores slightly below the softmax baseline [arxiv:2203.14335 tab. 6] |
| Need evidence outside dense segmentation | Nothing in this source | The paper evaluates segmentation only [arxiv:2203.14335 §4.1] |

The reported gains over DeepLabV3+ on Mapillary are 3.41 at the coarse level,
3.23 at the middle and 2.73 at the fine level [arxiv:2203.14335 tab. 1]. The
gains therefore shrink towards the finest labels. The same method also
reports gains on three further datasets and on three backbones
[arxiv:2203.14335 tab. 1, tab. 2, tab. 3, tab. 4]. All numbers are
author-reported.

## Implementation sketch

Training step, using the paper's equations. Items marked "optional" follow the
paper's ablations and are not needed for the focal tree-min variant.

```pseudocode
# Hierarchical head and loss, one training step  [§3.1] [§3.2] [§3.3]
# Input:  image batch, leaf labels, tree with parent links
# Output: scalar loss

def hierarchical_loss(images, leaf_labels, step):
    # Score every tree node with a sigmoid, internal nodes included  [§3.1]
    feats = encoder(images)
    scores = sigmoid(node_head(feats))            # (B, H, W, num_nodes)
    # Binary target per node: 1 if the leaf lies under that node  [§3.2.1]
    node_targets = ancestor_targets(leaf_labels, tree)
    # Tree-min probability: minimum over ancestors if the node is positive,
    # maximum over descendants if negative  [eq. 4]
    probs = where(node_targets,
                  min_over_ancestors(scores),
                  max_over_descendants(scores))
    # Focal binary cross-entropy with exponent 2  [eq. 6] [tab. 7]
    loss = focal_bce(probs, node_targets, gamma=2.0)
    # Optional: tree-triplet term on a 256-d projection  [§3.2.2] [§3.3]
    if use_triplet:
        emb = proj_head(feats)                    # (B, H, W, 256)
        # Weight rises on a cosine curve from 0 to 0.5  [§3.3]
        beta = 0.25 * (1 - cos(pi * step / total_steps))
        loss += beta * triplet_loss(emb, leaf_labels, tree)
    return loss
```

Inference picks the best root-to-leaf path. For a two-level tree, this is the
leaf whose own score plus its parent's score is largest [arxiv:2203.14335
eq. 3]:

```pseudocode
# Inference: best feasible root-to-leaf path  [§3.1] [eq. 3]
def predict(scores):
    # Sum node scores along the path from each leaf up to the root
    path_score = sum_along_path_to_root(scores)   # one value per leaf
    # Pick the leaf with the largest path score
    return argmax(path_score)
```

## Contradictions

None between papers, because the page has one source.

Within the one source, the text and the tables differ on small points. The
text gives gains over DeepLabV3+ of 2.69, 3.21 and 3.40 points [arxiv:2203.14335
§4.2]. Table 1 gives 2.73, 3.23 and 3.41 [arxiv:2203.14335 tab. 1]. This page
uses the table values. Section 3.2.1 says a gamma of 0 makes the focal loss
equal the tree-min loss. Table 7's gamma-0 row (84.47 at the coarse level) does
not match the tree-min row of Table 6 (83.48) [arxiv:2203.14335 §3.2.1, tab.
6, tab. 7].

## Pitfalls

- **Released code is partial.** The paper's ablation favours the focal
  tree-min loss [arxiv:2203.14335 tab. 6]. The released Cityscapes loss calls a
  non-focal version, although a focal version exists in the same file
  [code: Pytorch/mmseg/models/losses/hiera_triplet_loss_cityscape.py#L137]. The
  PASCAL and LIP path does use the focal loss [code:
  Pytorch/mmseg/models/losses/rmi_hiera_triplet_loss.py#L161]. The release has
  no Mapillary, OCRNet or MaskFormer configs, so the Transformer results in the
  paper cannot be reproduced from it. Check which loss a given config trains
  before comparing numbers.
- **Margin units.** The paper's distance is half of one minus cosine. A
  constant margin of 0.6 on one minus cosine is not the same as 0.6 on the
  paper's scale [arxiv:2203.14335 §3.2.2, eq. 7].
- **Triplet sampling.** The paper samples triplets from the batch but does not
  say how. Sampling choices change which pixels supply the triplets
  [arxiv:2203.14335 §3.2.2].
- **Loss normalisation.** The paper does not say whether the loss is summed or
  averaged over pixels and classes. This sets the relative weight of the
  metric term [arxiv:2203.14335 eq. 2, eq. 6].
- **Hardware.** The paper's timing is for its own hardware. The per-batch
  overhead depends on the setup [arxiv:2203.14335 tab. D.1].

## Papers

- [Deep Hierarchical Semantic Segmentation](/papers/2203.14335): proposes a
  per-node sigmoid head with a focal tree-min loss and a tree-triplet metric
  term, evaluated on four segmentation datasets.