Deep Hierarchical Semantic Segmentation

In plain words

Most segmentation models label every pixel with one flat class, such as "bicycle" or "pedestrian". Real categories nest, though. A bicyclist is a kind of rider, and a rider is a kind of human. HSSN lets each pixel carry its whole chain of labels, from the most general class down to the finest one. It does this by asking the model a yes-or-no question for every class in the tree, so a pixel can count as "human", "rider" and "bicyclist" at once.

Two extra training losses make those answers agree with the tree, so the model cannot call a pixel "bicyclist" without also calling it "human". A third loss pulls pixels of related classes closer together in the feature space than pixels of unrelated classes.

The approach works with existing segmentation networks and needs only a small change to the final output layer. The authors argue that it also makes some mistakes less costly: confusing a bicycle with a motorcycle is a smaller error than confusing a pedestrian with a lamppost, and the tree lets the model prefer the first kind of mistake.

TL;DR

HSSN turns a flat segmentation network into a hierarchical one by making the final head predict a sigmoid score for every node of a class tree, instead of one softmax over the leaves §3.1. Training adds a focal tree-min loss that enforces the tree's ancestor and descendant rules §3.2.1, eq. 6, and a tree-triplet loss with a margin set from tree distances, applied to a 256-d projection of the embeddings §3.2.2, §3.3. The paper reports gains at every hierarchy level on four datasets. The released code departs from the paper in several loss and training details, so read "Paper vs code" before reproducing numbers. The code also has no Mapillary, OCRNet or MaskFormer configs.

Problem

Standard segmentation predicts one of a set of disjoint classes per pixel and ignores how classes relate. The authors say earlier hierarchical segmentation methods either target human body parts only or change the architecture with graph message passing, and neither imposes tree-shaped label constraints on the prediction or the training loss §1. Their stated goals are (i) to adapt hierarchy-agnostic networks to the hierarchical setting cheaply, and (ii) to use the hierarchy as a regulariser during training §1.

The method assumes a tree-shaped hierarchy: each class has one parent, the root is the most general class, and the leaves are the finest classes §3.1. Every class is taken to be both its own ancestor and its own descendant §3.1. The paper does not say how to handle a class with more than one parent.

Method

Head and loss. The encoder maps an image to a dense feature map. In the baseline, a softmax over the leaf classes is trained with categorical cross-entropy eq. 1. HSSN instead applies a sigmoid to every node of the tree, including internal nodes, and trains with binary cross-entropy over all nodes eq. 2 §3.1. The sigmoid scores are per node and are not forced to sum to one.

Inference. For each pixel, HSSN picks the feasible root-to-leaf path with the largest sum of node scores eq. 3 §3.1. A feasible path starts at a leaf, ends at the root, and moves from each node to its parent §3.1.

Hierarchy constraints. The paper states two consistency rules. The positive rule: if a node is labelled positive, all its ancestors must be positive. The negative rule: if a node is labelled negative, all its descendants must be negative §3.2.1. These become score constraints. For a positive node, each ancestor score must be at least the node's score. For a negative node, each descendant score must be at most the node's score §3.2.1.

Tree-min probabilities. To satisfy both constraints by construction, each node's probability is taken from the tree rather than from the raw score. For a positive node, the probability is the minimum over its ancestors, including itself. For a negative node, it is the maximum over its descendants, including itself eq. 4 §3.2.1. The tree-min loss is the binary cross-entropy on these probabilities eq. 5. The probabilities satisfy both constraints by construction, so any violating prediction is penalised §3.2.1.

Focal tree-min loss. The paper adds a focal modulating factor with exponent gamma. It is applied as (1 − p)^gamma on positive terms and p^gamma on negative terms eq. 6. Gamma is set to 2 §3.3. With gamma = 0 the paper says the loss equals the tree-min loss §3.2.1.

Tree-triplet loss. The triplet loss is defined over pixel triplets (anchor, positive, negative) drawn from the training batch, such that the tree distance from anchor to positive is smaller than from anchor to negative §3.2.2. The tree distance is the number of edges on the shortest path between two nodes §3.2.2. The distance between embeddings is 0.5 times (1 − cosine similarity), so it lies in [0, 1] eq. 7 §3.2.2. The loss is a hinge on the difference of distances plus a margin eq. 7. The margin is 0.1 plus 0.5 times a term that depends on the tree distances and the tree height D eq. 8. The paper calls the constant 0.1 a tolerance for intra-class variance §3.2.2.

Projection and total objective. The triplet loss uses a projection head with two 1×1 convolutions and a ReLU between them, mapping each pixel embedding to 256 dimensions. The head is discarded after training and adds no inference cost §3.3. The total objective is the focal tree-min loss plus beta times the tree-triplet loss, where beta lies in [0, 0.5] and follows a cosine annealing schedule §3.3 app. C.

# Hierarchical segmentation training step  [§3.1] [§3.2] [§3.3]
# Input:  images (B, 3, H, W); leaf labels (B, H, W); tree with parents
# Output: scalar loss

def training_step(images, leaf_labels, step):
    # Encode pixels and score every tree node with a sigmoid  [§3.1]
    features = encoder(images)                       # (B, C, H', W')
    scores = sigmoid(seg_head(features))             # (B, H', W', |V|)
    # Binary target per node: 1 if the leaf lies under it  [§3.2.1]
    node_labels = ancestor_targets(leaf_labels)      # (B, H', W', |V|)
    # Tree-min: positive nodes take the minimum over their ancestors  [eq. 4]
    tree_min = min_over_ancestors(scores)            # includes the node itself
    # Tree-min: negative nodes take the maximum over their descendants  [eq. 4]
    tree_max = max_over_descendants(scores)          # includes the node itself
    probs = where(node_labels, tree_min, tree_max)   # (B, H', W', |V|)
    # Focal binary cross-entropy on the tree-min probabilities  [eq. 6]
    loss_seg = focal_bce(probs, node_labels, gamma=2.0)
    # Projection head used only by the triplet loss  [§3.3]
    embeds = proj_head(features)                     # (B, H', W', 256)
    # Sample anchor, positive and negative pixels from the batch  [§3.2.2]
    triplets = sample_triplets(leaf_labels, tree, count=200)   # [app. D]
    # Embedding distance is 1/2 (1 - cosine similarity)  [§3.2.2] [eq. 7]
    d_pos = half_cosine_distance(embeds, triplets.anchor, triplets.pos)
    d_neg = half_cosine_distance(embeds, triplets.anchor, triplets.neg)
    # Hierarchy-induced margin from tree distances, D = tree height  [eq. 8]
    margin = 0.1 + 0.5 * (triplets.tree_dist_neg
                          - triplets.tree_dist_pos) / (2 * tree_height)
    # Hinge on the distance gap, averaged over triplets  [eq. 7]
    loss_tt = relu(d_pos - d_neg + margin).mean()
    # Beta: cosine from 0 to 0.5; this form comes from code, not the paper
    # [§3.3] [code: see Paper vs code, beta schedule]
    beta = 0.25 * (1 - cos(pi * step / total_steps))
    return loss_seg + beta * loss_tt

Equations as stated in the paper §3.1 §3.2 §3.3:

\mathcal{L}^{\text{CCE}}(\bm{y}) = -\log(y_{\hat{v}_\chi}) eq. 1

\mathcal{L}^{\text{BCE}}(\bm{s}) = \sum_{v \in \mathcal{V}} -\hat{l}_v \log(s_v) - (1-\hat{l}_v)\log(1-s_v) eq. 2

\{v^*_1, \dots, v^*_{|\mathcal{P}|}\} = \arg\max_{\mathcal{P} \subseteq \mathcal{T}} \sum_{v_p \in \mathcal{P}} s_{v_p} eq. 3

p_v = \min_{u \in \mathcal{A}_v} s_u \ \text{if}\ \hat{l}_v = 1; \qquad 1 - p_v = \min_{u \in \mathcal{C}_v}(1 - s_u) = 1 - \max_{u \in \mathcal{C}_v} s_u \ \text{if}\ \hat{l}_v = 0 eq. 4

\mathcal{L}^{\text{TM}}(\bm{p}) = \sum_{v \in \mathcal{V}} -\hat{l}_v \log p_v - (1-\hat{l}_v)\log(1-p_v) eq. 5

\mathcal{L}^{\text{FTM}}(\bm{p}) = \sum_{v \in \mathcal{V}} -\hat{l}_v (1-p_v)^{\gamma} \log p_v - (1-\hat{l}_v) p_v^{\gamma} \log(1-p_v) eq. 6

\mathcal{L}^{\text{TT}}(\bm{i}, \bm{i}^+, \bm{i}^-) = \max\{\langle \bm{i}, \bm{i}^+\rangle - \langle \bm{i}, \bm{i}^-\rangle + m,\ 0\}, \quad \langle \bm{x}, \bm{y} \rangle = \tfrac{1}{2}\left(1 - \tfrac{\bm{x}\cdot\bm{y}}{\|\bm{x}\|\|\bm{y}\|}\right) eq. 7

m = m_\varepsilon + 0.5\, m_\tau, \qquad m_\tau = \frac{\psi(\hat{v}_\chi, \hat{v}^-_\chi) - \psi(\hat{v}_\chi, \hat{v}^+_\chi)}{2D}, \qquad m_\varepsilon = 0.1 eq. 8

In these equations, the leaf label of pixel i is written v̂ (the paper's \hat{v}_\chi), the binary label of node v is \hat{l}_v, A_v and C_v are the ancestor and descendant sets of v (both include v), and ψ is the tree distance.

Hyperparameters

Results

Author-reported. Setup: four datasets (Mapillary Vistas 2.0, Cityscapes, LIP, PASCAL-Person-Part). Backbones: ResNet-101, HRNetV2-W48, Swin-Small. Segmentation heads: DeepLabV3+, OCRNet, MaskFormer §4.1 §4.2. Each row reports mIoU per hierarchy level, where the dataset has the level. The superscripts are not defined in the paper. Inferred from the magnitudes, mIoU^1 is the finest (leaf) level and mIoU^3 the coarsest.

Mapillary Vistas 2.0, validation set tab. 1

Cityscapes, validation set tab. 2. Only mIoU^2 and mIoU^1 are reported.

PASCAL-Person-Part, test set tab. 3. Abridged: the per-part columns (head, torso, arms, legs, body groups, background) are omitted here. All models use ResNet-101.

LIP, validation set tab. 4. All models use ResNet-101 unless marked.

Ablations on Mapillary Vistas 2.0 (val) and PASCAL-Person-Part (test), ResNet-101.

Essential components tab. 5: FTM is the focal tree-min loss; TT is the tree-triplet loss.

Loss variants tab. 6:

Gamma sweep tab. 7:

Tree-triplet variants tab. 8:

Distance measure tab. 9:

Appendix hyperparameter ablations on Cityscapes validation tab. C.1:

Number of triplets tab. D.1. Time is the training time per batch in seconds. The row marked "--" has no triplet loss.

Paper vs code

Official repository: https://github.com/0liliulei/HieraSeg (linked in the paper p. 1), cloned at commit cc3c1cfbabe3cc2af620e0193a245822cca8a841. The repository has no LICENSE file at the root or in its subdirectories. The code lives in Pytorch/ (a fork of the mmseg framework) and PaddlePaddle/ (README only). Released configs cover DeepLabV3+ with ResNet-101 for Cityscapes, PASCAL-Person-Part and LIP only. The Mapillary dataset config exists, but no Mapillary HSSN config is present. No OCRNet, MaskFormer or Swin configs appear in the tree.

• Cityscapes uses the non-focal tree-min loss. The paper's best loss is the focal tree-min eq. 6 tab. 6. The Cityscapes loss calls losses_hiera, which has no focal term, although a focal version exists in the same file hiera_triplet_loss_cityscape.py hiera_triplet_loss_cityscape.py. The PASCAL and LIP path does use the focal loss rmi_hiera_triplet_loss.py.
• Extra losses not in the paper. Cityscapes adds cross-entropy on the 19 leaf outputs and on the 7 super-class outputs hiera_triplet_loss_cityscape.py. PASCAL and LIP add a region mutual information (RMI) term at weight 0.5 and a focal term at weight 0.5, plus cross-entropy on the leaves and two group levels rmi_hiera_triplet_loss.py. The base model also has an auxiliary FCN head with loss weight 0.4 [code: Pytorch/configs/base/models/deeplabv3plus_r50-d8_vd_contrast.py#L32-L43].
• Loss scale. Each tree level is averaged over valid pixels and over the number of classes, not summed as in eq. 2 and eq. 6. The Cityscapes losses are then multiplied by 5 hiera_triplet_loss_cityscape.py. The PASCAL and LIP focal loss has no factor 5 hiera_loss.py.
• Triplet margin is a constant, and the distance differs. The code uses 1 − cosine, not 0.5 times (1 − cosine). It uses a constant margin of 0.6, with a comment that says so tree_triplet_loss.py. The paper's margin is 0.1 + 0.5 m_τ, which is dynamic eq. 8. On the paper's distance scale, the code's 0.6 corresponds to 0.3.
• Triplet positives and negatives. The code takes as positives the other leaves under the anchor's super-class, and as negatives the leaves outside it. It excludes the anchor's own class tree_triplet_loss.py. The paper defines positives by smaller tree distance §3.2.2.
• Triplet sampling is deterministic. The code takes the first min(anchors, positives, negatives, 200) indices per class in flattened batch order, not a random sample tree_triplet_loss.py. Triplet losses are averaged over the classes present tree_triplet_loss.py.
• Projection head. The code takes the embedding from the last backbone stage, adds a normalisation layer, and L2-normalises the output sep_aspp_contrast_head.py sep_aspp_contrast_head.py. The paper mentions neither.
• Beta schedule. The code's beta is 0.25 (1 − cos(π t / T)) with T = 80,000 hard-coded hiera_triplet_loss_cityscape.py. The PASCAL and LIP code sets T to 60,000 or 160,000 rmi_hiera_triplet_loss.py. Paper: cosine annealing on [0, 0.5], with no stated formula §3.3.
• Inference. The code adds each super-class score to its leaves, drops the super-class outputs, and takes the argmax over leaves encoder_decoder.py. This matches the path score in eq. 3 for a two-level path. The code has no root output.
• Class hierarchy. The paper's Cityscapes text says 6 super-classes §4.1. The code defines 7 hiera_triplet_loss_cityscape.py. Appendix A says a virtual root "All" is added for Cityscapes app. A. The code's output count is 26 = 19 + 7, so it has no root node.
• PASCAL-Person-Part classes. The paper says both "6 human parts" and "20 fine-grained parts" §4.1. The code's PASCAL loss has 7 leaf labels (ids 0 to 6), with upper ids 1 to 4 and lower ids 5 to 6 rmi_hiera_triplet_loss.py. LIP has 20 leaves (19 parts plus background) rmi_hiera_triplet_loss.py.
• Schedule. The PASCAL config runs 60K iterations deeplabv3plus_r101-d8_480x480_60k_pascal_person_part_hiera_triplet.py. The paper says 80K §4.1.
• Learning rate and weight decay. PASCAL and LIP use base LR 7e-3, and every config uses weight decay 5e-5 [code: Pytorch/configs/base/schedules/schedule_60k.py#L2] [code: Pytorch/configs/base/schedules/schedule_160k.py#L2]. The paper says 1e-2 and 1e-4 §4.1.
• Testing. The paper uses six scales with flip §4.1. The default test config has the scale list commented out and flip off [code: Pytorch/configs/base/datasets/cityscapes.py#L23-L25]. The README's --aug-test option depends on that list README.MD.
• Augmentation. The code adds photometric distortion and cat_max_ratio 0.75 [code: Pytorch/configs/base/datasets/cityscapes.py#L11-L13]. Neither is in the paper.
• Batch size. The paper gives batch 8 and 16 §4.1. The configs give samples per GPU, and the GPU count is not stated anywhere in the release.
• Hard-coded values. Schedule lengths are hard-coded in both loss files, so changing max_iters needs a code edit hiera_triplet_loss_cityscape.py rmi_hiera_triplet_loss.py. The triplet code calls .cuda() directly tree_triplet_loss.py, so it runs only on GPU.

No discrepancies were found for the focal gamma (2 in both paper and code), the sigmoid over all nodes, the 200-triplet cap per class, or the random scale and crop ranges.

Pitfalls

• Paper and code disagree on the loss actually trained. Cityscapes trains a non-focal tree-min loss plus extra cross-entropy terms, while the paper's ablation favours focal tree-min tab. 6. Reproduce the code's loss if you want its numbers, and the paper's loss if you want the method as described. The two give different results.
• Margin units. A margin of 0.6 on the paper's distance scale is not the same as 0.6 on 1 − cosine. Use the paper's scale, or double the code's margin.
• γ = 0 is inconsistent. Section 3.2.1 says gamma = 0 makes the focal loss equal the tree-min loss §3.2.1. Table 7's gamma = 0 row (84.47 / 70.24 / 39.52 on Mapillary) differs from the TM row of Table 6 (83.48 / 69.13 / 38.69) tab. 6 tab. 7.
• Text and table disagree on Mapillary. The text gives gains over DeepLabV3+ of 2.69, 3.21 and 3.40 points §4.2. Table 1 gives 2.73, 3.23 and 3.41 tab. 1. The OCRNet and MaskFormer deltas match the table.
• Cityscapes super-class count. The text says 6 §4.1. The code and the appendix hierarchy use 7 app. A hiera_triplet_loss_cityscape.py. Check the hierarchy before reusing the leaf-to-group mapping.
• PASCAL split and class count. The paper's ablation and Table 3 say the test set tab. 3, while Appendix B.3 says val app. B. Its §4.1 text gives both 6 and 20 parts.
• Triplet cost. Table D.1 shows the time per batch rising from 0.93 to 1.18 seconds on Mapillary, about 27% more, when 200 triplets are added tab. D.1. This is the paper's own figure, and the cost depends on hardware.
• Anchor sampling bias. The code takes the first N pixels of each class in flattened batch order, so the first image(s) containing a class supply its triplets tree_triplet_loss.py. The paper says triplets are sampled from the batch, without saying how §3.2.2.
• Label resolution. The code resizes labels to the feature-map size with nearest-neighbour interpolation before sampling triplets tree_triplet_loss.py. The paper does not say at what resolution sampling happens.
• Unstated normalisation. The paper does not say whether the tree-min loss is summed over pixels, averaged over pixels, or normalised per class. The code averages per level and multiplies by 5 for Cityscapes. This sets the relative weight of the triplet term.
• Undefined superscript. The paper does not define mIoU^1, mIoU^2 and mIoU^3. Compare the number of levels with the dataset before reading across tables.
• Released code is partial. Only DeepLabV3+ on ResNet-101 is released, with no Mapillary, OCRNet, MaskFormer or Swin configs. Transformer results in tab. 1 and tab. 2 are not reproducible from this release.
• No licence. The repository has no LICENSE file, so the code cannot be copied or quoted. The card cites it only.

Lineage

• Builds on: focal loss, adapted to tree-min probabilities §3.2.1 [lin2017focal]; triplet loss with hierarchy-aware sampling and margin §3.2.2 [schroff2015facenet]; hierarchical classification constraints of the positive T-property §3.2.1 [bi2011multilabel].
• Compared against: DeepLabV3+, OCRNet, MaskFormer, HMSANet, HHP, SemaTree, CNIF and BGNet, all on the same datasets §4.2.
• Superseded by: not stated in this wiki.