wikillm

Concept · 2 papers · updated 2026-11-15

Heatmap regression

Predict a dense map and read each answer from its peak. Pose variants use one map per keypoint; a detection variant uses per-class centre maps on a LiDAR grid. One pose card adds a training-only geometry and edge target. Choose by the task, whether ground-truth joints exist, the inference budget and the decoding.

computer-visionpose-estimationkeypointsheatmapsobject-detection

Markdown7 min read

AI-compiled overview, updated 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

Heatmap regression outputs a dense spatial map and reads the answer off its peak. In pose estimation the map has one channel per keypoint. It is high near the keypoint and low elsewhere, and the keypoint is read off the peak. The training target for each map is a Gaussian peak at the joint, and the loss is the squared error between predicted and target maps, summed over joints 1705.02407 §3.4 1705.02407 eq. 7. In the card's network the output is 15 maps, 14 joints plus background, at 64 × 64 for a 256 × 256 input crop 1705.02407 §3.2 1705.02407 §3.4.

In LiDAR detection the map is a bird's-eye score map with one channel per class, and a peak marks a likely object centre rather than a joint. Its training loss is a focal loss, not a squared error 2210.07372 §3.5 2210.07372 eq. 6.

The invariant the variants share is that a peak in a dense map, rather than a direct coordinate regression, gives the answer. Variants differ in what the map indexes (keypoint or class), what extra signal shapes the maps during training, and how the peaks are picked out at test time.

Variants

The first two variants are for pose estimation; the third is for detection.

Plain heatmap regression

The same network trained with the heatmap loss alone. The paper's ablation includes this row without the knowledge target, which reaches 92.6 PCK@0.2 on the LSP test set against 93.9 with it 1705.02407 tab. 1. The paper also lists its own reimplementation of the stacked hourglass at 91.5 on the same set, which is not a published number 1705.02407 tab. 1.

Knowledge-guided heatmap regression

A side branch exists only during training. It is built from the ground-truth joints 1705.02407 §3.3:

A learned linear projection maps the heatmaps to this target, with an L2 regulariser on its weights 1705.02407 eq. 5. The projection losses are weighted 0.05 1705.02407 §3.4. The training objective mixes the projection loss and the heatmap loss with a weight λ that decays over training 1705.02407 eq. 8 1705.02407 alg. 1. At test time the branch and the projection are removed, so the deployed network has no extra layers 1705.02407 §3.4 1705.02407 alg. 2. The backbone is a fractal network of hourglass blocks built from inception-residual units 1705.02407 §3.1 1705.02407 §3.2. Card: Knowledge-guided fractal pose network.

Object-centre heatmaps in LiDAR detection

The detector on the card SWFormer puts a CenterNet-style head on a bird's-eye voxel grid. Its heatmap has one channel per class and is trained with a penalty-reduced focal loss, with ε = 1e-3, α = 2 and β = 4 2210.07372 §3.5 2210.07372 eq. 6. Voxels with a foreground score above γ = 0.05 are scattered onto a dense grid for voxel diffusion, which spreads features into empty cells near object centres 2210.07372 §3.4. Box regression runs on voxels whose ground-truth heatmap value is above δ₁ = 0.2 2210.07372 §4.2 2210.07372 eq. 9. The paper does not state the ground-truth heatmap formula and refers to RSN for it 2210.07372 §3.5.

The diffusion step matters in the authors' ablation. With kernel k = 9, vehicle 3D L1 AP is 78.50; with k = 1, which means no diffusion, it is 72.13 2210.07372 tab. 4.

Decoding and test-time steps

These steps sit on top of a heatmap model. Each is a separate choice from the training variant.

  • Peak picking (pose). The paper replaces the per-map maximum with a cross-heatmap rule: take the strongest blob, remove other blobs of the same map and nearby blobs in any map, and repeat 1705.02407 §3.5. The ablation names it 3D-NMS and credits it with +0.5 points on LSP 1705.02407 §5.4 1705.02407 tab. 1.
  • Flip and scale averaging (pose). The paper predicts on the original and the mirrored crop, at scales 1 and 0.75, un-mirrors and swaps left and right maps, then averages the maps joint by joint 1705.02407 §5.2.3 1705.02407 alg. 2. The ablation credits flipping with +0.7 and scaling with +0.7 on LSP 1705.02407 §5.4.
  • Local maxima without NMS (detection). The head finds local maxima of each class map with max pooling, keeps those at or above δ₂ = 0.1, and applies no non-maximum suppression afterwards 2210.07372 §3.5 2210.07372 §4.2. The max-pool kernel size is not stated 2210.07372 §3.5. The ablations in tab. 4 and tab. 5 do not test this choice.

When to use which

SituationUseWhy
Pose: ground-truth joints exist for the training images and extra training cost is acceptableKnowledge-guided variantThe paper reports 93.9 against 92.6 for the same network without the target on LSP PCK@0.2 1705.02407 tab. 1. The branch is removed at test time 1705.02407 §3.4.
Pose: no joint-level annotation to build the geometry and edge targets from, or the training pipeline cannot add a side branchPlain heatmap regressionThe target is built from ground-truth joints 1705.02407 §3.3, so the variant cannot run without them.
Pose: tight inference budgetPlain heatmap regression with no test-time averaging, or a lighter backboneThe paper's network runs at 62 ms per image against 47 ms for its stacked hourglass baseline on one Pascal TITAN X 1705.02407 §4. Flip and scale add forward passes 1705.02407 alg. 2.
Pose: benchmark-style reporting on MPII or LSPKnowledge-guided variant with the full test-time stepsThe paper reports 91.2 PCKh@0.5 on MPII test and 93.9 PCK@0.2 on LSP test with these steps 1705.02407 tab. 2 1705.02407 tab. 3. These are author-reported.
Detection: single-stage LiDAR detection with per-class centre scores on a bird's-eye gridObject-centre variantThe paper reports 73.36 mAPH/L2 on the Waymo test set, above the methods in its test table 2210.07372 tab. 3. These are author-reported. The per-class map is decoded without NMS 2210.07372 §3.5.
Detection: a latency budget on a different stackCheck before adoptingThe only latency figure in the card is 43 ms, for XLA on one Nvidia T4 2210.07372 §4.3.

The pose gains are for one network on two datasets. The LSP ablation is the only place the components are separated, and the MPII test ground truth is not public, so the ablation is LSP-only 1705.02407 §5.4. The authors also say that testing at scale 0.75 on MPII "may not be ideal", because cropping cut off some feet and lowered the ankle scores 1705.02407 §5.3.1.

The detection numbers come from one network on the Waymo Open Dataset. The ablations are on vehicle and pedestrian AP only 2210.07372 tab. 4 2210.07372 tab. 5. Several details needed to re-implement it are not stated, including batch size, the number of head layers and the ground-truth heatmap formula 2210.07372 §3.5 2210.07372 §4.2. The card reports no official code.

Implementation sketch

The common core is the target, the loss and the decoder. The knowledge branch is an addition on top.

Pseudocode
# 1. Crop the person and warp to 256 x 256  [arxiv:1705.02407 §3.2]
# 2. Predict 15 maps at 64 x 64: 14 joints plus background  [arxiv:1705.02407 §3.4]
def forward(crop, network):
    return network(crop)                       # (15, 64, 64)

# 3. Target: one Gaussian peak per joint  [arxiv:1705.02407 §3.4]
def targets(joints):
    maps = [gaussian_peak(j, sigma) for j in joints]
    maps.append(background_map(joints))        # 20 : 1 fg/bg weight [§5.2.2]
    return stack(maps)

# 4. Heatmap loss: squared error summed over joints  [arxiv:1705.02407 eq. 7]
def heatmap_loss(pred, target):
    return sum(squared_error(pred[j], target[j]) for j in range(14))

# 5. Training only: knowledge target from ground-truth joints  [arxiv:1705.02407 §3.3]
def knowledge_target(image, joints):
    return geometry_features(joints), edge_features(image, joints)

# 6. Training only: projection of the maps, L2 weight 0.05  [arxiv:1705.02407 §3.4]
#    with the L2 regulariser on W  [arxiv:1705.02407 eq. 5]
def projection_loss(pred, knowledge, w_proj):
    return squared_error(linear(w_proj, pred), knowledge) + l2(w_proj)

# 7. Training only: mix the two losses with a decaying weight  [arxiv:1705.02407 eq. 8]
def total_loss(lam, kp_loss, hm_loss):
    return lam * kp_loss + (1 - lam) * hm_loss

# 8. Decode: strongest blob, suppress neighbours, repeat  [arxiv:1705.02407 §3.5]
def decode(maps):
    return cross_heatmap_nms(maps)             # one (x, y) per joint

Notes: the Hough and HOG functions in steps 5 and 6 are described in 1705.02407 §3.3; the paper does not give the exact feature layout, so they are placeholders here. The paper also leaves β, the λ schedule and the normalisation ε unstated 1705.02407 eq. 4 1705.02407 eq. 5 1705.02407 eq. 8.

For the detection variant, steps 3 and 4 become a per-class focal loss over a bird's-eye map, and step 8 becomes the max-pool decoder with a cut at δ₂ = 0.1 and no NMS 2210.07372 §3.5. The ground-truth heatmap target is not stated in the card, so it is left out here.

Contradictions

None across cards on the same quantity. The two papers measure different things: keypoint PCK on 2D person crops 1705.02407 tab. 1 and 3D LiDAR AP and mAPH on Waymo 2210.07372 tab. 3. Their numbers are not comparable. Inside the pose card, the stacked hourglass row is the authors' own reimplementation and is not a published number 1705.02407 tab. 1.

Papers

  • Knowledge-guided fractal pose network: the knowledge-guided heatmap variant, its LSP ablation and the MPII and LSP results.
  • SWFormer: the object-centre heatmap variant with voxel diffusion and NMS-free decoding for LiDAR detection.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.