---
kind: concept
slug: heatmap-regression
title: Heatmap regression
summary: >-
  Predict a dense map and read each answer from its peak. Pose variants use one map per keypoint; a detection variant uses per-class centre maps on a LiDAR grid. One pose card adds a training-only geometry and edge target. Choose by the task, whether ground-truth joints exist, the inference budget and the decoding.
tags: [computer-vision, pose-estimation, keypoints, heatmaps, object-detection]
papers: ["1705.02407", "2210.07372"]
related: [multi-person-pose-estimation, unsupervised-keypoint-detection, knowledge-injection, test-time-augmentation, object-detection, focal-loss, bev-representation]
status: draft
updated_at: 2026-11-15
---

## Overview

Heatmap regression outputs a dense spatial map and reads the answer off its
peak. In pose estimation the map has one channel per keypoint. It is high near
the keypoint and low elsewhere, and the keypoint is read off the peak. The
training target for each map is a Gaussian peak at the joint, and the loss is
the squared error between predicted and target maps, summed over joints
[arxiv:1705.02407 §3.4] [arxiv:1705.02407 eq. 7]. In the card's network the
output is 15 maps, 14 joints plus background, at 64 × 64 for a 256 × 256 input
crop [arxiv:1705.02407 §3.2] [arxiv:1705.02407 §3.4].

In LiDAR detection the map is a bird's-eye score map with one channel per
class, and a peak marks a likely object centre rather than a joint. Its
training loss is a focal loss, not a squared error [arxiv:2210.07372 §3.5]
[arxiv:2210.07372 eq. 6].

The invariant the variants share is that a peak in a dense map, rather than a
direct coordinate regression, gives the answer. Variants differ in what the
map indexes (keypoint or class), what extra signal shapes the maps during
training, and how the peaks are picked out at test time.

## Variants

The first two variants are for pose estimation; the third is for detection.

### Plain heatmap regression

The same network trained with the heatmap loss alone. The paper's ablation
includes this row without the knowledge target, which reaches 92.6 PCK@0.2 on
the LSP test set against 93.9 with it [arxiv:1705.02407 tab. 1]. The paper also
lists its own reimplementation of the stacked hourglass at 91.5 on the same
set, which is not a published number [arxiv:1705.02407 tab. 1].

### Knowledge-guided heatmap regression

A side branch exists only during training. It is built from the ground-truth
joints [arxiv:1705.02407 §3.3]:

- **Geometry target.** For each pair of adjacent joints, the line through them
  is written in Hough coordinates, and the target holds the normalised line
  vector, the joint positions and the number of visible joints on that line
  [arxiv:1705.02407 §3.3] [arxiv:1705.02407 eq. 1].
- **Edge target.** HOG descriptors are computed along each limb with 8
  orientation bins and L2 block normalisation [arxiv:1705.02407 §3.3]
  [arxiv:1705.02407 eq. 2] [arxiv:1705.02407 eq. 3] [arxiv:1705.02407 eq. 4].
  Features of self- or object-occluded joints are set to zero
  [arxiv:1705.02407 §3.3].

A learned linear projection maps the heatmaps to this target, with an L2
regulariser on its weights [arxiv:1705.02407 eq. 5]. The projection losses are
weighted 0.05 [arxiv:1705.02407 §3.4]. The training objective mixes the
projection loss and the heatmap loss with a weight λ that decays over training
[arxiv:1705.02407 eq. 8] [arxiv:1705.02407 alg. 1]. At test time the branch and
the projection are removed, so the deployed network has no extra layers
[arxiv:1705.02407 §3.4] [arxiv:1705.02407 alg. 2]. The backbone is a fractal
network of hourglass blocks built from inception-residual units
[arxiv:1705.02407 §3.1] [arxiv:1705.02407 §3.2]. Card:
[Knowledge-guided fractal pose network](/papers/1705.02407).

### Object-centre heatmaps in LiDAR detection

The detector on the card [SWFormer](/papers/2210.07372) puts a CenterNet-style head on
a bird's-eye voxel grid. Its heatmap has one channel per class and is trained
with a penalty-reduced focal loss, with ε = 1e-3, α = 2 and β = 4
[arxiv:2210.07372 §3.5] [arxiv:2210.07372 eq. 6]. Voxels with a foreground
score above γ = 0.05 are scattered onto a dense grid for voxel diffusion, which
spreads features into empty cells near object centres [arxiv:2210.07372 §3.4].
Box regression runs on voxels whose ground-truth heatmap value is above
δ₁ = 0.2 [arxiv:2210.07372 §4.2] [arxiv:2210.07372 eq. 9]. The paper does not
state the ground-truth heatmap formula and refers to RSN for it
[arxiv:2210.07372 §3.5].

The diffusion step matters in the authors' ablation. With kernel k = 9, vehicle
3D L1 AP is 78.50; with k = 1, which means no diffusion, it is 72.13
[arxiv:2210.07372 tab. 4].

## Decoding and test-time steps

These steps sit on top of a heatmap model. Each is a separate choice from the
training variant.

- **Peak picking (pose).** The paper replaces the per-map maximum with a
  cross-heatmap rule: take the strongest blob, remove other blobs of the same
  map and nearby blobs in any map, and repeat [arxiv:1705.02407 §3.5]. The
  ablation names it 3D-NMS and credits it with +0.5 points on LSP
  [arxiv:1705.02407 §5.4] [arxiv:1705.02407 tab. 1].
- **Flip and scale averaging (pose).** The paper predicts on the original and the
  mirrored crop, at scales 1 and 0.75, un-mirrors and swaps left and right
  maps, then averages the maps joint by joint [arxiv:1705.02407 §5.2.3]
  [arxiv:1705.02407 alg. 2]. The ablation credits flipping with +0.7 and
  scaling with +0.7 on LSP [arxiv:1705.02407 §5.4].
- **Local maxima without NMS (detection).** The head finds local maxima of
  each class map with max pooling, keeps those at or above δ₂ = 0.1, and applies
  no non-maximum suppression afterwards [arxiv:2210.07372 §3.5]
  [arxiv:2210.07372 §4.2]. The max-pool kernel size is not stated
  [arxiv:2210.07372 §3.5]. The ablations in tab. 4 and tab. 5 do not test this
  choice.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Pose: ground-truth joints exist for the training images and extra training cost is acceptable | Knowledge-guided variant | The paper reports 93.9 against 92.6 for the same network without the target on LSP PCK@0.2 [arxiv:1705.02407 tab. 1]. The branch is removed at test time [arxiv:1705.02407 §3.4]. |
| Pose: no joint-level annotation to build the geometry and edge targets from, or the training pipeline cannot add a side branch | Plain heatmap regression | The target is built from ground-truth joints [arxiv:1705.02407 §3.3], so the variant cannot run without them. |
| Pose: tight inference budget | Plain heatmap regression with no test-time averaging, or a lighter backbone | The paper's network runs at 62 ms per image against 47 ms for its stacked hourglass baseline on one Pascal TITAN X [arxiv:1705.02407 §4]. Flip and scale add forward passes [arxiv:1705.02407 alg. 2]. |
| Pose: benchmark-style reporting on MPII or LSP | Knowledge-guided variant with the full test-time steps | The paper reports 91.2 PCKh@0.5 on MPII test and 93.9 PCK@0.2 on LSP test with these steps [arxiv:1705.02407 tab. 2] [arxiv:1705.02407 tab. 3]. These are author-reported. |
| Detection: single-stage LiDAR detection with per-class centre scores on a bird's-eye grid | Object-centre variant | The paper reports 73.36 mAPH/L2 on the Waymo test set, above the methods in its test table [arxiv:2210.07372 tab. 3]. These are author-reported. The per-class map is decoded without NMS [arxiv:2210.07372 §3.5]. |
| Detection: a latency budget on a different stack | Check before adopting | The only latency figure in the card is 43 ms, for XLA on one Nvidia T4 [arxiv:2210.07372 §4.3]. |

The pose gains are for one network on two datasets. The LSP ablation is the
only place the components are separated, and the MPII test ground truth is not
public, so the ablation is LSP-only [arxiv:1705.02407 §5.4]. The authors also
say that testing at scale 0.75 on MPII "may not be ideal", because cropping cut
off some feet and lowered the ankle scores [arxiv:1705.02407 §5.3.1].

The detection numbers come from one network on the Waymo Open Dataset. The
ablations are on vehicle and pedestrian AP only [arxiv:2210.07372 tab. 4]
[arxiv:2210.07372 tab. 5]. Several details needed to re-implement it are not
stated, including batch size, the number of head layers and the
ground-truth heatmap formula [arxiv:2210.07372 §3.5] [arxiv:2210.07372 §4.2].
The card reports no official code.

## Implementation sketch

The common core is the target, the loss and the decoder. The knowledge branch
is an addition on top.

```pseudocode
# 1. Crop the person and warp to 256 x 256  [arxiv:1705.02407 §3.2]
# 2. Predict 15 maps at 64 x 64: 14 joints plus background  [arxiv:1705.02407 §3.4]
def forward(crop, network):
    return network(crop)                       # (15, 64, 64)

# 3. Target: one Gaussian peak per joint  [arxiv:1705.02407 §3.4]
def targets(joints):
    maps = [gaussian_peak(j, sigma) for j in joints]
    maps.append(background_map(joints))        # 20 : 1 fg/bg weight [§5.2.2]
    return stack(maps)

# 4. Heatmap loss: squared error summed over joints  [arxiv:1705.02407 eq. 7]
def heatmap_loss(pred, target):
    return sum(squared_error(pred[j], target[j]) for j in range(14))

# 5. Training only: knowledge target from ground-truth joints  [arxiv:1705.02407 §3.3]
def knowledge_target(image, joints):
    return geometry_features(joints), edge_features(image, joints)

# 6. Training only: projection of the maps, L2 weight 0.05  [arxiv:1705.02407 §3.4]
#    with the L2 regulariser on W  [arxiv:1705.02407 eq. 5]
def projection_loss(pred, knowledge, w_proj):
    return squared_error(linear(w_proj, pred), knowledge) + l2(w_proj)

# 7. Training only: mix the two losses with a decaying weight  [arxiv:1705.02407 eq. 8]
def total_loss(lam, kp_loss, hm_loss):
    return lam * kp_loss + (1 - lam) * hm_loss

# 8. Decode: strongest blob, suppress neighbours, repeat  [arxiv:1705.02407 §3.5]
def decode(maps):
    return cross_heatmap_nms(maps)             # one (x, y) per joint
```

Notes: the Hough and HOG functions in steps 5 and 6 are described in
[arxiv:1705.02407 §3.3]; the paper does not give the exact feature layout, so
they are placeholders here. The paper also leaves β, the λ schedule and the
normalisation ε unstated [arxiv:1705.02407 eq. 4] [arxiv:1705.02407 eq. 5]
[arxiv:1705.02407 eq. 8].

For the detection variant, steps 3 and 4 become a per-class focal loss over a
bird's-eye map, and step 8 becomes the max-pool decoder with a cut at δ₂ = 0.1
and no NMS [arxiv:2210.07372 §3.5]. The ground-truth heatmap target is not
stated in the card, so it is left out here.

## Contradictions

None across cards on the same quantity. The two papers measure different things:
keypoint PCK on 2D person crops [arxiv:1705.02407 tab. 1] and 3D LiDAR AP and
mAPH on Waymo [arxiv:2210.07372 tab. 3]. Their numbers are not comparable.
Inside the pose card, the stacked hourglass row is the authors' own
reimplementation and is not a published number [arxiv:1705.02407 tab. 1].

## Papers

- [Knowledge-guided fractal pose network](/papers/1705.02407): the knowledge-guided heatmap variant, its LSP ablation and the MPII and LSP results.
- [SWFormer](/papers/2210.07372): the object-centre heatmap variant with voxel diffusion and NMS-free decoding for LiDAR detection.
