Multi-person pose estimation

Overview

The output is a set of people. Each person has a box, a score and a list of body keypoints such as wrists, knees and nose, and each keypoint must belong to exactly one person. The benchmark used in both source cards is COCO keypoints. The YOLO-Pose card scores with Object Keypoint Similarity (OKS) and reports AP 2204.06806 §4.1; the Lightweight OpenPose card reports AP on COCO validation 1811.12004 §3.2.

The invariant all variants keep is that the keypoints of each person are attached to that person. The families differ in where that attachment happens. Bottom-up methods find keypoints across the whole image and group them after the fact. Top-down methods detect people first and estimate a pose inside each box. Single-stage methods predict the pose at the same time as the box, so the grouping is built into the output 2204.06806 §1, §2.1, §2.2 1811.12004 §1, §2.

Variants

Bottom-up keypoints with grouping (lightweight OpenPose)

Bottom-up methods predict keypoints and limb fields for the whole image in one pass, then assemble people from them. Their cost does not grow with the number of people, which the source card gives as their main advantage 1811.12004 §2. The grouping step is the expensive part on a CPU: in the paper's profile of the baseline, grouping ran fast, but resizing feature maps and extracting keypoints did not 1811.12004 tab. 3.

Lightweight OpenPose keeps the OpenPose approach and makes each part lighter 1811.12004 §4:

• A dilated MobileNet v1 backbone through conv5_5, with output stride 8 1811.12004 §4.1.1.
• A neck where three depthwise separable convolutions reduce the channels from 256 to 128 1811.12004 §4.1.2 tab. 5.
• Refinement blocks made of a 1x1 convolution, a 3x3 convolution and a dilated 3x3 convolution, with a residual connection. The paper reports about 2.5 times fewer operations than the 7x7 convolution they replace 1811.12004 §4.1.2.
• Faster post-processing, where the upsampling factor trades accuracy against speed 1811.12004 §4.2.

Author-reported: 9 GFLOPs and 42.8 AP on COCO validation, against 61.7 GFLOPs and 43.4 AP for the 2-stage OpenPose baseline 1811.12004 tab. 2 tab. 5. The paper does not label the 43.4 row as the baseline; the card matches it by GFLOPs. End-to-end speed is 28 fps on an Intel NUC and 26 fps on an i7-6850K, measured on a video with more than 20 poses with OpenVINO 1811.12004 tab. 6 §4.3. Card: Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose.

Its training values are not stated in the paper, which points to the original OpenPose training parameters 1811.12004 §4.1.

Single-stage keypoints on a detector head (YOLO-Pose)

A YOLOv5-style detector gets a second output per anchor. Each anchor predicts a box, a class score, and 17 COCO keypoints with a confidence for each. The keypoints of a person are stored with the anchor that matched that person, so no grouping step is needed 2204.06806 §3, §3.2. The keypoint regression is trained with a loss built from OKS, which the authors chose so that training targets the evaluation metric 2204.06806 §3.4. Card: YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss.

Reported result: 90.2 AP50 on COCO val2017 and 90.3 AP50 on test-dev2017 for the largest model, with no test-time augmentation. AP on the same rows is 69.4 and 68.5 2204.06806 tab. 1, tab. 2. Model sizes run from YOLOv5s6-pose (15.1M parameters, 22.8 GMACS at 960 input) to YOLOv5l6-pose (87.0M, 145.6 GMACS) 2204.06806 tab. 1.

Top-down detect-then-estimate

Named as background only. The source card says these methods run a person detector and then a pose network per person, so their cost grows with the number of people 2204.06806 §2.1. No card here measures them beyond the authors' comparison rows in 2204.06806 tab. 1, tab. 2.

When to use which

The source cards are the only evidence on this page, so the table only claims what they report. The authors' own results are not independently checked.

Implementation sketch

Two cores are shown: bottom-up grouping from limb fields, and the single-stage inference and loss. Each step carries its paper citation.

# Bottom-up grouping from keypoint maps and limb fields  [arxiv:1811.12004 §3.1]
# Input:  heatmaps (H, W, K), limb fields (H, W, 2L) at 1/8 input scale
# Output: people, each a set of keypoints

def group_people(heatmaps, limb_fields, up):
    # Upsample before reading peaks; the factor trades speed for accuracy  [§4.2]
    heatmaps = upsample(heatmaps, up)
    limb_fields = upsample(limb_fields, up)
    # Candidate keypoints are local peaks of the heatmaps  [§3.1]
    peaks = find_peaks(heatmaps)
    connections = []
    for limb in limb_types:
        # Score each candidate pair by how well the field points along the limb  [§3.1]
        # Sample count and cut-offs are not stated in the paper
        connections += score_pairs(peaks, limb_fields, limb)
    # Merge connections that share a keypoint into people  [§3.1]
    return merge_into_people(connections)

# Single-stage pose inference  [arxiv:2204.06806 §3] [eq. 1] [fig. 2]
# Input:  image (3, H, W)
# Output: per person a box, a class score and 17 keypoints (x, y, conf)

def predict_people(image):
    # Backbone and neck give feature maps at several scales  [§3.1] [fig. 2]
    feature_maps = neck(backbone(image))
    # Each anchor predicts 6 box values and 51 keypoint values  [eq. 1]
    raw = [head(fmap) for fmap in feature_maps]
    # Boxes go through standard object-detection post-processing  [§1]
    people = detection_postprocess(raw)
    for person in people:
        # Keep keypoints whose confidence is above 0.5  [§3]
        person.keypoints = [
            kpt for kpt in person.keypoints if kpt.conf > 0.5
        ]
    return people

# Training loss for one matched anchor  [eq. 2] [eq. 3] [eq. 4] [eq. 5]
# Summed over all matched scales, anchors and locations  [eq. 5]

def pose_loss(pred, target):
    # Box term: 1 minus CIoU  [eq. 2]
    box_loss = 1 - ciou(pred.box, target.box)
    # Keypoint term: 1 minus OKS over labelled keypoints  [eq. 3] [§3.4]
    kpt_loss = 1 - oks(pred.kpts, target.kpts, target.labelled)
    # Keypoint confidence: binary cross-entropy against visibility  [eq. 4]
    kpt_conf_loss = bce(pred.kpt_conf, target.labelled)
    # Classification term; its form is not stated in the paper  [eq. 5]
    cls_loss = classification_loss(pred.cls, target.cls)
    # Weights 0.5, 0.05, 0.1 and 0.5  [eq. 5] [§3.4]
    return (
        0.5 * cls_loss
        + 0.05 * box_loss
        + 0.1 * kpt_loss
        + 0.5 * kpt_conf_loss
    )

Where variants differ: the decoding of keypoints, the OKS scale and the per-keypoint weights. The single-stage paper encodes keypoints relative to the anchor centre without normalisation 2204.06806 §3.2, and its per-keypoint constants k_n are not given 2204.06806 §3.4. The bottom-up paper gives its grouping thresholds only in code, not in the text 1811.12004 §3.1. Check the cards' pitfalls before porting.

Contradictions

• Headline metric. The YOLO-Pose abstract and discussion lead with AP50. On AP, several bottom-up rows are higher, for example HigherHRNet W48 at 640 with 69.9 on val2017 2204.06806 tab. 1. Both numbers are reported by the authors, so compare on matching metrics.
• Compute units and matching. The YOLO-Pose paper reports GMACS and the Lightweight OpenPose paper reports GFLOPs 2204.06806 tab. 1 1811.12004 tab. 2. Convert before comparing. The YOLO-Pose authors claim about 4× less compute for high AP50, citing YOLOv5m6-pose against DEKR W48 on test-dev 2204.06806 §1 tab. 2. Their tables mark flip and multi-scale test rows with multiplied complexity, and those rows are only comparable with each other 2204.06806 tab. 1.
• Accuracy across papers. Lightweight OpenPose reports 42.8 AP on a COCO validation subset at a final input of 456 x 256 1811.12004 §3.2 §4.3. YOLO-Pose reports 63.8 AP on val2017 at 960 input [arxiv:2204.06806 tab. 1]. The subsets, inputs and models differ, so the two numbers are not comparable.
• Reproduced accuracy. The Lightweight OpenPose card records that the released checkpoint reports about 40% AP, not the paper's 42.8. See the Paper vs code section of that card.
• Paper and code. The YOLO-Pose paper's keypoint head is on YOLOv5. The code that holds keypoint training is in a YOLOX repository, with a different decode, a different loss weighting and a 640 input, not the 960 input of the tables 2204.06806 §3.1 fig. 2. This page does not treat the code as the paper's method.
• Loss baseline. The YOLO-Pose authors say the L1 baseline needed loss-weight tuning and OKS did not 2204.06806 §4.3. The ablation in 2204.06806 tab. 3 may therefore favour the OKS row.

Papers

• Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose: bottom-up keypoints and limb fields with a lighter backbone, neck and refinement, for CPU real time.
• YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss: single-stage keypoints on a detector head, trained with an OKS-based keypoint loss.