wikillm

Paper · 2017-05-05 · cs.CV

Knowledge-Guided Deep Fractal Neural Networks for Human Pose Estimation

Pose estimation on person-centred crops with a fractal network of nested hourglass blocks built from inception-residual units. It predicts 15 heatmaps (14 joints plus background). Training adds an auxiliary loss that ties the heatmaps to limb geometry and HOG edge features from ground truth; the branch is dropped at test time. Author-reported: PCK@0.2 93.9 on LSP, PCKh@0.5 91.2 on MPII.

Heatmap regressionKnowledge injectionTest-time augmentationpose-estimationkeypointsheatmapsknowledge-guided-learningcomputer-vision

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

The task is to find the body joints (head, shoulders, elbows, wrists, hips, knees, ankles) of a person in a picture. The usual approach is a deep network that outputs one "heat map" per joint, where a bright spot marks where that joint probably is. Such networks learn appearance well but do not know that joints sit at fixed distances and angles from each other, so they can link a joint to the wrong limb.

The paper's idea is to teach that body structure during training only. While the network trains, a side branch looks at the true joint positions and builds two kinds of description: how each pair of neighbouring joints lies in the image, and what the edges along each limb look like. The network is pushed so that its heat maps can be converted into these descriptions. When training ends, the side branch is thrown away, so the deployed network is no bigger.

The network itself stacks small hourglass-shaped blocks, each built from several convolution paths that are merged in two ways. Test-time tricks (flipping the picture, a second scale, and a custom rule that picks the joint points out of the heat maps) add a little more accuracy. The authors report state-of-the-art results on two standard benchmarks.

TL;DR

  • 2-D pose estimation on person-centred crops, output as 15 heatmaps (14 joints plus background) §3.4 §5.2.2.
  • Backbone: a four-level hourglass whose blocks are inception-residual modules, arranged as a "fractal" network §3.1 §3.2.
  • Training-only knowledge branch: Hough-style line features and HOG edge features from ground truth, regressed from the heatmaps by a learned projection with an auxiliary L2 loss §3.3 §3.4. Removed at test time §3.4.
  • Author-reported: PCKh@0.5 91.2 on MPII test and PCK@0.2 93.9 on LSP test, both called state of the art §5.3.

Problem

Heatmap regression for pose predicts each joint without an explicit model of the body, so the joint geometry and inter-joint dependence must be learned from data alone. The authors argue that such coupled geometric constraints are hard for a plain network to capture and propose to inject them as a training-time prior §1 §3.3.

Method

Network §3.1 §3.2 fig. 2 fig. 4 fig. 6

  1. Input: a 256 × 256 crop, downsampled to 64 × 64 for all modules and for the output heatmaps §3.2.
  2. Basic block (inception-resnet): two branches are fused by channel-wise concatenation, which a 1 × 1 convolution then mixes, and by pixel-wise addition in the resnet style. Convolutions keep spatial size §3.2 fig. 4.
  3. Improved hourglass: one branch is downsampled and then upsampled, while the other keeps full resolution; the two are added pixel-wise. The hourglass is recursive over four levels, and the lowest level is an inception-resnet block §3.2 fig. 6.
  4. Fractal: the same configuration appears at every level, so the network is nested inside itself §3.1.
  5. Output: 15 heatmaps, 14 joints plus one background map §3.4.

Knowledge branch (training only) §3.3 §3.4

  1. For each pair of adjacent joints, the line through them is written in Hough coordinates (θ, ρ) eq. 1 §3.3.
  2. Each line is also described by its normalised vector, the joint positions, and the number of visible joints on the line §3.3.
  3. HOG descriptors are computed around each limb: horizontal and vertical gradient filters, gradient magnitude and orientation eq. 2 eq. 3; 8 orientation bins; L2 block normalisation eq. 4 §3.3. Limbs are located from the training metadata, not by a sliding window §3.3.
  4. Self- or object-occluded joints have their features set to zero §3.3.
  5. Geometry features are regressed from the 8-channel heatmap projection into a 224-dimensional vector. Edge features are regressed from a separate branch. Each projection is trained with an L2 loss weighted 0.05 §3.4.
  6. A learned linear projection W maps the heatmaps to the knowledge target K, with L2 regularisation on W eq. 5 §3.4.

Losses and training §3.4 §4 alg. 1

  1. Ground-truth heatmaps: one Gaussian peak per joint §3.4.
  2. Heatmap loss over joints eq. 7. The overall objective mixes the projection loss and the heatmap loss with a weight λ that decays during training eq. 8. The projection gradient flows back into the network eq. 9 §3.4.
  3. At test time the knowledge branch and the projection are removed §3.4 alg. 2.

Cross-heatmap NMS §3.5 fig. 8

  1. Instead of taking the pixel maximum in each heatmap, the method finds blobs of high response in every heatmap. It takes the strongest blob, removes other blobs of the same heatmap and nearby blobs of any heatmap, and repeats until none remain §3.5.

The equations, in the order they appear in the paper:

Hough line parameters for a joint pair eq. 1:

\theta = \arctan\left(\frac{x_i - x_j}{y_j - y_i}\right), \quad \rho = x_j \cos\theta + y_j \sin\theta eq. 1

HOG gradient magnitude and orientation eq. 2 eq. 3:

|G| = \sqrt{I_x^2 + I_y^2} eq. 2

\varphi = \arctan\frac{I_y}{I_x} eq. 3

HOG block normalisation as printed eq. 4:

f = \frac{v}{\sqrt{\lVert v \rVert_2^2} + e^2} eq. 4

Knowledge-projection loss eq. 5:

\mathcal{L}_{KP} = \lVert K - W H_J \rVert_2^2 + \beta \lVert W \rVert_2^2 eq. 5

Heatmap as belief per joint eq. 6:

H_j(p) \leftarrow S_j(Y_j = p) eq. 6

Heatmap loss eq. 7:

\mathcal{L}_f = \sum_{j \in J} \lVert H_j(p) - H_j^*(p) \rVert_2^2 eq. 7

Joint objective eq. 8:

W_f^* \leftarrow \arg\min_{W_f} \left(\lambda \mathcal{L}_{KP} + (1 - \lambda) \mathcal{L}_f\right) eq. 8

Gradient applied to the injection layer eq. 9:

\Delta W_{f,i} = -\lambda \frac{\partial \mathcal{L}_{KP}}{\partial W_{f,i}} eq. 9

Training, as summarised by the paper:

Pseudocode
# Training phase  [alg. 1]  [§4]
# Input:  RGB images I; ground-truth joints J
# Output: fractal network weights W_f; projection weights W_KP

def train(images, joints, epochs):
    # 1. Repeat for k epochs, over mini-batches  [alg. 1]
    for epoch in range(epochs):
        for batch_images, batch_joints in batches(images, joints):
            # 2. Build the knowledge target K from images and joints  [§3.3]
            knowledge = encode_knowledge(batch_images, batch_joints)
            # 3. Backpropagate to W_f and W_KP  [alg. 1]
            loss = mix_losses(knowledge, batch_images, batch_joints)
            update(loss)  # mix weight lambda decays  [eq. 8] [§3.4]
    return fractal_weights, projection_weights

Testing, as summarised by the paper:

Pseudocode
# Testing phase  [alg. 2]  [§5.2.3]
# Input:  cropped person image (3, 256, 256); trained fractal network
# Output: one (x, y) point per joint, in image coordinates

def predict_joints(image, network):
    # 1. Predict on the original and flipped image at scales 1 and 0.75
    #    [§5.2.3]
    heatmap_sets = []
    for scale in [1.0, 0.75]:
        scaled = resize(image, scale)
        heatmap_sets.append(network_heatmaps(scaled))
        flipped = network_heatmaps(mirror(scaled))
        # 2. Un-mirror the maps and swap left and right limbs  [§5.2.3]
        #    [code: testing/utils/utils_pose.py#L150-L160]
        heatmap_sets.append(swap_left_right(mirror(flipped)))
    # 3. Average the heatmap sets, joint by joint  [§5.2.3]
    merged = average(heatmap_sets)
    # 4. Pick joint points with cross-heatmap NMS  [§3.5]
    return cross_heatmap_nms(merged)

Hyperparameters

NameValueWhereNotes
input crop256 × 256§3.2 §5.2.1warped from a patch centred on the target person
output heatmaps15 (14 joints + background), 64 × 64§3.4 §3.2
hourglass levels4, recursive§3.2first level is an inception-resnet block
network depthabout 250 conv layers in the original hourglass; over 300 in this network§4
geometry projection1 × 1 conv to 8 channels, then FC to 224§3.4
edge projection1 × 1 convs halving channels down to 32, then FC§3.4FC width not stated
projection layers2 FC + 3 conv in total§3.4 §4removed at test time
projection loss weight0.05 for geometry; 0.05 for edges§3.4see Paper vs code
β (L2 on W)not statedeq. 5
λ (mix weight)not stated; decays during trainingeq. 8 §3.4schedule not stated
HOG orientation bins8§3.3
HOG ε (block normalisation)not statedeq. 4 §3.3
foreground : background loss weight20 : 1§5.2.2
optimiserRMSprop§5.2.2
initial learning rate2.5 × 10⁻⁴§5.2.2halved on validation plateau
minimum learning rate10⁻⁶§5.2.2
batch sizenot stated§5.2.2see Paper vs code
training lengthover 300 epochs on MPII + extended LSP§5.2.24 Pascal TITAN GPUs
rotation augmentation±30°§5.2.1
rescale augmentation0.75 to 1.25§5.2.1
flip and colour jitterflip and colour jitter used§5.2.1
test scales1 and 0.75, original and flipped§5.2.3
validation splitTompson's split of MPII§5.2.2
initialisationnot stated—
test time per image62 ms (single Pascal TITAN X)§4 §5.2.3stacked hourglass: 47 ms §4

Results

Author-reported. Setup: MPII test set (PCKh@0.5) and extended LSP test set (PCK@0.2 and PCP@0.5). The same model is used for both sets, trained on MPII plus extended LSP training data with person-centric annotations §5.2.2 §5.3. Baseline numbers are as listed in the paper's tables; the paper cites each method.

MPII test, PCKh@0.5 tab. 2

MethodHeadSho.Elb.Wri.HipKneeAnk.Total
Ours98.196.392.287.890.687.682.791.2
Newell et al., ECCV'1698.296.391.287.190.187.483.690.9
Bulat & Tzimiropoulos, ECCV'1697.995.189.985.389.485.781.789.7
Wei et al., CVPR'1697.895.088.784.088.482.879.488.5
Insafutdinov et al., ECCV'1696.895.289.384.488.483.478.088.5
Rafi et al., BMVC'1697.293.986.481.386.880.673.486.3
Gkioxary et al., ECCV'1696.293.186.782.185.281.474.186.1
Lifshitz et al., ECCV'1697.893.385.780.485.376.670.285.0
Pishchulin et al., CVPR'1694.190.283.477.382.675.768.682.4
Hu & Ramanan, CVPR'1695.091.683.076.681.974.569.582.4
Tompson et al., CVPR'1596.191.983.977.880.972.364.882.0
Carreira et al., CVPR'1695.791.781.772.482.873.266.481.3
Tompson et al., NIPS'1495.890.380.574.377.669.762.879.6
Pishchulin et al., ICCV'1374.349.040.834.136.534.435.244.1

AUC on MPII: 63.6 §5.3.1.

LSP test, PCK@0.2 tab. 3

MethodHeadSho.Elb.Wri.HipKneeAnk.Total
Ours98.294.491.889.394.795.093.593.9
Bulat & Tzimiropoulos, ECCV'1697.292.188.185.292.291.488.790.7
Wei et al., CVPR'1697.892.587.083.991.590.889.990.5
Insafutdinov et al., ECCV'1697.492.787.584.491.589.987.290.1
Pishchulin et al., CVPR'1697.091.083.878.191.086.782.087.1
Lifshitz et al., ECCV'1696.889.082.779.190.986.082.586.7
Belagiannis & Zisserman, FG'1795.289.081.577.083.787.082.885.2
Yu et al., ECCV'1687.288.282.476.391.485.878.784.3
Rafi et al., BMVC'1695.886.279.375.086.683.879.883.8
Yang et al., CVPR'1690.678.173.868.874.869.958.973.6
Chen & Yuille, NIPS'1491.878.271.865.573.370.263.473.4
Fan et al., CVPR'1592.475.265.364.076.768.370.473.0
Tompson et al., NIPS'1490.679.267.963.469.571.064.272.3
Pishchulin et al., ICCV'1387.256.746.738.061.057.552.757.1
Wang & Li et al., CVPR'1384.757.143.736.756.752.450.854.6

AUC on LSP: 69.1 §5.3.2.

LSP test, PCP@0.5 tab. 4

MethodTorsoU.LegL.LegU.ArmForearmHeadTotal
Ours98.695.893.690.784.296.492.3
Bulat & Tzimiropoulos, ECCV'1697.792.489.386.779.795.288.9
Wei et al., CVPR'1698.082.289.185.877.995.088.3
Insafutdinov et al., ECCV'1697.090.686.986.179.595.487.8
Yu et al., ECCV'1698.093.188.182.972.683.085.4
Pishchulin et al., CVPR'1697.088.882.082.471.895.884.3
Lifshitz et al., ECCV'1697.388.884.480.671.494.884.3
Belagiannis & Zisserman, FG'1796.086.782.279.469.489.482.1
Rafi et al., BMVC'1697.687.380.276.866.293.381.2
Yang et al., CVPR'1695.678.571.872.261.883.974.8
Chen & Yuille, NIPS'1496.077.272.269.758.185.673.6
Fan et al., CVPR'1595.477.769.862.849.186.670.1
Tompson et al., NIPS'1490.370.461.163.051.283.766.6
Pishchulin et al., ICCV'1388.763.658.446.035.285.158.0
Wang & Li et al., CVPR'1387.556.055.843.132.179.154.1

Ablation, LSP PCK@0.2 tab. 1 §5.4. The "Hourglass" row is the authors' own implementation of the stacked hourglass inside their system, because it was not reported on LSP.

MethodHeadSho.Elb.Wri.HipKneeAnk.Total
Hourglass97.093.088.885.692.293.090.991.5
Ours (no guidance)97.993.289.186.494.593.892.992.6
Ours (with guidance)98.294.491.889.394.795.093.593.9
Plain testing97.492.788.886.792.293.892.292.0
+ flipping97.793.390.487.593.294.292.892.7
+ scaling98.193.791.388.794.094.693.293.4
+ 3D-NMS98.294.491.889.394.795.093.593.9

From the text §5.4: inception-resnet over the plain resnet block gives +1.1 points; knowledge guidance gives +1.3; flipping +0.7; scaling +0.7; cross-heatmap NMS +0.5. The table rows agree with these gains.

Caveats the authors give: MPII ground truth for the test set is not public, so the ablation is LSP-only §5.4. Testing at scales 1 and 0.75 on MPII "may not be ideal" because cropping cut off some feet, which lowers ankle scores §5.3.1.

Paper vs code

Code: the repository linked in the paper, https://github.com/Guanghan/GNet-pose, at commit c70e0fc65b290e68a16ca3040a70300f9c2bee44. The copyright line in the root LICENSE.md names the first author, Guanghan Ning LICENSE.md. The root LICENSE.md opens with Apache-2.0 but its body restricts use to academic or non-profit noncommercial research. We treat the code as non-commercial and cite it only: no code is quoted, and the notes below describe it in our own words.

Matches:

  • The heatmap and background loss weighting, 1 for joints and 0.05 for background, gives the 20 : 1 ratio stated in the paper §5.2.2 gen_GNet.py.
  • Geometry branch: 8 channels, a 224-dimensional fully connected layer and a 0.05 loss weight §3.4 gen_GNet.py.
  • Crop 256, rotation 30 degrees, rescale 0.75 to 1.25 §5.2.1 data_inject.prototxt; stride 4 gives the 64 × 64 output §3.2 data_inject.prototxt.
  • Test-time flip with left/right order swap, scales 1 and 0.75, and NMS on §5.2.3 test.py utils_pose.py.
  • The deployed network has no injection or loss layers GNet_deploy.prototxt.

Discrepancies:

  • Edge loss weight: the paper says 0.05 §3.4; the code uses 0.005 gen_GNet.py GNet_train.prototxt.
  • Geometry input: the paper says the ground-truth heatmap is convolved §3.4; the code convolves the predicted background heatmap gen_GNet.py.
  • Edge branch: the paper takes it from a third inception-residual module and halves channels down to 32 §3.4. The code concatenates the predicted joint heatmaps with the image pooled to 64 × 64, then applies 64- and 32-channel convolutions and a 64-wide FC gen_GNet.py.
  • Mix weight: the paper weights the projection and heatmap losses with a λ that decays during training eq. 8 §3.4. The code has no λ and uses fixed loss weights gen_GNet.py gen_GNet.py.
  • Learning rate: the paper gives RMSprop at 2.5 × 10⁻⁴, halved on plateau down to 10⁻⁶ §5.2.2. The public solver is RMSProp with a fixed base rate of 10⁻⁶ GNet_solver.prototxt. The README dates an update of the prototxts and model to 9/27/2018 README.md.
  • Training length: the paper gives over 300 epochs and no batch size §5.2.2. The code uses batch size 2 data_inject.prototxt and max_iter 490000 GNet_solver.prototxt. The epoch count cannot be checked from the repository.
  • Knowledge features: the Hough and HOG encoders of §3.3 are not in this repository. The data layer declares the injected blobs data_inject.prototxt, but that layer belongs to a modified Caffe (linked from the README, not included) README.md. Neither genLMDB.py nor the other scripts compute them. These steps are unverified.
  • Augmentation: the paper lists flips and colour jitter §5.2.1; the data-layer config does not set either data_inject.prototxt.
  • Reproducibility: the README says the LSP 93.9 result came after a newer Caffe fixed an up/downsampling issue, and the paper notes its PyCaffe build may not match the Torch hourglass numbers on MPII §5.4 README.md. The README also reports a later, unpublished model at 94.0071 on LSP.
  • NMS detail: the paper describes blob suppression in general terms §3.5. The code runs an OpenCV blob detector on an 8-bit inverted heatmap, ignores the background map, suppresses blobs whose centres lie within the mean of their diameters, and falls back to the heatmap minimum when a joint has no blob utils_nms.py utils_nms.py utils_nms.py utils_nms.py.

Pitfalls

  • Printed HOG normalisation (eq. 4): the denominator is written as √‖v‖² + e², which is probably meant to be √(‖v‖² + e²). The value of e is not stated. The code does not contain this step (see Paper vs code).
  • Hough angle (eq. 1): the formula uses arctan of a ratio, which is undefined for horizontal limbs (y_j = y_i). The quadrant rule is not stated.
  • Undefined constants: β in eq. 5, the λ schedule in eq. 8, and the HOG ε are not stated.
  • Ground truth or prediction: §3.4 says the ground-truth heatmap feeds the projection, while the code feeds the prediction. Pick one and test it. The code also uses the predicted maps for the edge branch.
  • Geometry vector layout: the 224-dimensional vector is described as normalised line vectors, joint positions and visibility counts §3.3. Its exact order and length are not stated.
  • Missing batch size and init: neither batch size nor weight initialisation is stated §5.2.2.
  • Background labels: regions that are not the target person's keypoints, including other people's limbs, count as background §5.2.2. The paper expects other people's limbs to get lower responses as a result §5.2.2.
  • Occlusion masking: self- and object-occluded joints have zeroed knowledge features §3.3. Check that the masks are applied before the projection.
  • Cost: the knowledge branch and projection add training cost only. Inference is 62 ms against 47 ms for the stacked hourglass §4. The network has over 300 conv layers §4.
  • Naming: the paper calls the NMS "cross-heatmap" in §3.5 and "3D-NMS" in the text and table 1 §3.4 tab. 1.
  • Crop source: the inputs are person-centred crops or ROIs §5.2.2 §5.2.3. The paper does not describe a detector that produces them. For MPII it uses the given rough position and scale; for LSP, the image size as scale and the image centre as position §5.2.3.

Lineage

  • Builds on: stacked hourglass networks (Newell et al., ECCV'16), which this network modifies with inception-resnet blocks and a recursive design §3.2. Inception-residual ideas are taken from Szegedy et al. §3.1.
  • Knowledge guidance: injects prior knowledge through an auxiliary projection loss during training only §3.3 §3.4.
  • Compared against: the MPII and LSP baselines in tab. 2 to tab. 4, and the authors' own stacked hourglass in tab. 1.
  • Superseded by: not stated in the wiki.

Source and license

Adapted from “Knowledge-Guided Deep Fractal Neural Networks for Human Pose Estimation” by Guanghan Ning, Zhi Zhang, Zhihai He (arXiv:1705.02407), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Guanghan/GNet-pose at commit c70e0fc65b29 (Academic/non-profit non-commercial research use only (LICENSE.md; its header also names Apache-2.0)). Any quoted code is shown under that license.