YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss

In plain words

Bottom-up pose methods first find keypoints (wrists, knees, nose and so on) across the whole image, then work out which keypoints belong to which person. That grouping step is hard to speed up on small devices. Top-down methods first find each person and then estimate a pose inside each box, which means running the model once per person.

YOLO-Pose takes a third route. It is built on the YOLOv5 object detector, which already draws one box per object. The same detector also predicts a set of body points for each box, so a person's box and pose come out together, and the points are already grouped by person. Nothing has to be matched afterwards.

The authors also changed the training target. The standard pose metric is OKS, a similarity score, while common training losses are plain distances. Here the keypoint loss is built from OKS itself, so training pushes towards the score the benchmark uses.

TL;DR

YOLO-Pose extends YOLOv5 with a keypoint head that predicts 17 COCO keypoints per anchor alongside the box, so detection and pose come from one forward pass with no grouping step §3, eq. 1. Its keypoint loss is 1 minus Object Keypoint Similarity (OKS), so training optimises the evaluation metric directly [§3.4, eq. 3]. Author-reported: 90.2 AP50 on COCO val2017 and 90.3 AP50 on test-dev2017 for the largest model, with no test-time augmentation tab. 1, tab. 2. On AP, the largest model is not ahead of every bottom-up row tab. 1.

Problem

Bottom-up methods find all keypoints in one shot and then group them into people. Their heatmap post-processing (NMS, refinement, grouping) is not differentiable, so it cannot be trained end to end, and it is hard to speed up on CNN accelerators §2.2. Their training usually uses an L1 loss on heatmaps or offsets, which the authors argue is not equivalent to maximising OKS §1, §3.4. Top-down methods run a person detector and then a pose network per person, so their cost grows linearly with the number of people §2.1. The authors therefore target a fixed-cost, end-to-end, grouping-free model that works with standard object-detection post-processing §1. They limit complexity to 150 GMACS and focus on real-time models §3.

Method

Overview. The model is a single-shot detector based on YOLOv5. The backbone is CSP-Darknet53 and the neck is PANet, followed by detection heads at several scales. Each head branches into a box output and a keypoint output §3.1, fig. 2.

1. Prediction vector per anchor. Each anchor predicts a box (centre, width, height, box confidence, class confidence) and, for each of the 17 keypoints, an x, a y and a keypoint confidence. That is 6 box values and 51 keypoint values §3, eq. 1.
2. Anchors store the whole pose. A ground-truth person matched to an anchor keeps its full 2-D pose, so the keypoints of a person are attached to that person's box without a grouping step §3.2.
3. Anchor-relative encoding. Box centre is taken relative to the anchor centre, and box width and height are normalised by the anchor's width and height. Keypoint locations are also taken relative to the anchor centre but are not normalised §3.2.
4. Box loss. The box is trained with 1 minus CIoU between the predicted and ground-truth box §3.3, eq. 2. The authors argue that OKS behaves like a distance-IoU loss on keypoints and is scale-invariant §3.4.
5. Keypoint regression loss. For each labelled keypoint n, the squared Euclidean distance to the ground truth is passed through a Gaussian-like similarity that uses the object's scale s and a per-keypoint weight k_n. The similarity is averaged over labelled keypoints, and the loss is 1 minus that average §3.4, eq. 3. Keypoints on the face weigh more than body keypoints, and the weights are taken from the COCO authors §3.4.
6. Keypoint confidence loss. Each keypoint has a confidence trained with binary cross-entropy. Its target is 1 when the keypoint is visible or occluded (visibility flag greater than 0), and 0 when it lies outside the field of view §3, eq. 4.
7. Total loss. The per-anchor terms are summed over all matched scales, anchors and locations, with weights for the classification, box, keypoint and keypoint confidence terms §3.4, eq. 5.
8. Inference. A detection is kept by standard post-processing §1. Keypoints with confidence above 0.5 are kept and the rest are rejected. Keypoints outside the field of view are trained towards confidence 0, so this step removes them §3.
9. Deployment. The authors state that the model uses only standard operators and exports end to end to ONNX, run with ONNX Runtime §3.7.

# YOLO-Pose inference  [§3] [eq. 1] [fig. 2]
# Input:  image (3, H, W)
# Output: per person a box, a class score and 17 keypoints (x, y, conf)

def predict_people(image):
    # Backbone and PAN neck give feature maps at several scales  [§3.3] [fig. 2]
    feature_maps = pan_neck(backbone(image))
    # Each anchor predicts 6 box values and 51 keypoint values  [eq. 1]
    raw = [head(fmap) for fmap in feature_maps]
    # Boxes go through standard object-detection post-processing  [§1]
    people = detection_postprocess(raw)
    for person in people:
        # Keep keypoints with confidence above 0.5, reject the rest  [§3]
        person.keypoints = [
            kpt for kpt in person.keypoints if kpt.conf > 0.5
        ]
    return people

# Target encoding relative to the matched anchor  [§3.2]
# Input:  gt_box (centre, size), gt_kpts (17, 2), anchor (centre, size)
# Output: box offsets, box size ratios, keypoint offsets

def encode_targets(gt_box, gt_kpts, anchor):
    # Box centre relative to anchor centre; exact scaling not stated  [§3.2]
    box_xy = gt_box.centre - anchor.centre
    # Box width and height normalised by the anchor's size  [§3.2]
    box_wh = gt_box.size / anchor.size
    # Keypoints relative to anchor centre, not normalised  [§3.2]
    kpts = gt_kpts - anchor.centre
    return box_xy, box_wh, kpts

# Object Keypoint Similarity for one object  [eq. 3] [§3.4]
# pred, gt: (17, 2); labelled: (17,) with labelled = visibility > 0
# object_area: s^2 of the object; k: per-keypoint weights (values not stated)

def oks(pred, gt, labelled, object_area, k):
    # Squared Euclidean distance d_n^2 per keypoint  [eq. 3]
    squared_dist = sum((pred - gt) ** 2, axis=-1)
    # Similarity exp(-d_n^2 / (2 s^2 k_n^2)) per keypoint  [eq. 3]
    similarity = exp(-squared_dist / (2 * object_area * k ** 2))
    # Average over labelled keypoints only  [eq. 3]
    return sum(similarity * labelled) / sum(labelled)

# Training loss for one matched anchor  [eq. 2] [eq. 4] [eq. 5]
# Summed over all matched scales, anchors and locations  [eq. 5]

def yolo_pose_loss(pred, target):
    # Box term: 1 - CIoU between predicted and ground-truth box  [eq. 2]
    box_loss = 1 - ciou(pred.box, target.box)
    # Keypoint term: 1 - OKS over labelled keypoints  [eq. 3]
    kpt_loss = 1 - oks(
        pred.kpts, target.kpts, target.labelled, target.area, k
    )
    # Keypoint confidence: BCE against "labelled"  [eq. 4]
    kpt_conf_loss = bce(pred.kpt_conf, target.labelled)
    # Classification term; its form is not stated in the paper  [eq. 5]
    cls_loss = classification_loss(pred.cls, target.cls)
    # Weighted sum with weights 0.5, 0.05, 0.1, 0.5  [eq. 5] [§3.4]
    return (
        0.5 * cls_loss
        + 0.05 * box_loss
        + 0.1 * kpt_loss
        + 0.5 * kpt_conf_loss
    )

Hyperparameters

Results

Author-reported. Setup: COCO 2017 keypoint task, OKS-based AP. The paper says the keypoint dataset has over 200,000 images and 250,000 person instances, with train2017 at 57K images, val2017 at 5K and test-dev2017 at 20K §4.1. Complexity in the tables is GMACS. Rows marked † use flip test or multi-scale test, with complexity multiplied by 2× or 5.25× tab. 1 caption. YOLO-Pose rows use no test-time augmentation §3.5.

Table 1: COCO 2017 val2017 (author-reported) tab. 1

Table 2: COCO 2017 test-dev2017 (author-reported) tab. 2

Table 3: Loss ablation on YOLOv5-s6, input 960 (author-reported) tab. 3

Input resolution ablation, YOLOv5-s6 (author-reported) §4.4

Chosen base resolution is 960 §4.4. This table has no number or caption in the paper.

Table 4: Low-complexity models, under 30 GMACS, val2017 (author-reported) tab. 4

Table 5: Quantisation of YOLOv5-s6_960_relu, val2017 (author-reported) tab. 5

Figures (described, not reproduced):

• fig. 1: a crowded COCO val2017 image. Top: YOLOv5m6-pose, where keypoints stay with the right person. Bottom: HigherHRNet-W32, where grouping errors appear.
• fig. 2: the architecture. CSP-Darknet53 backbone, PANet neck over four scales (P3 to P6), then detection heads, each splitting into a box output and a keypoint output.
• fig. 3: top, a keypoint outside the box is predicted correctly; bottom, two riders with a loose box, where a top-down method would miss the ankle.
• fig. 4: mAP against GMACS for low-complexity models, below 30 GMACS.
• fig. 5 and fig. 6: AP50 against GMACS on val2017 and test-dev2017, with YOLO-Pose points plotted alongside the other methods.

Paper vs code

Two official repositories are linked in the abstract, both under the TexasInstruments organisation. The authors are affiliated with Texas Instruments Inc. p. 1. Both were cloned at HEAD and searched for keypoint and OKS code.

• edgeai-yolox (code on this card; commit cbdd9d8, BSD-3-Clause and Apache-2.0 per its LICENSE file): this holds the human-pose training code. The paper's model is anchor-based YOLOv5, but the pose experiment here is YOLOX-s, an anchor-free detector with depth=0.33 and width=0.50 yolox_s_human_pose.py. It uses a PAFPN neck over three input scales with 256, 512 and 1024 channels yolox_s_human_pose.py, and one anchor per location yolo_kpts_head.py. The paper describes four scales and three anchors §3.3 fig. 2.
• edgeai-yolov5 (commit 74e9db5, GPL-3.0, copyleft, so cited by path only): this holds YOLOv5 but no keypoint loss or keypoint head at this commit. It does hold a pretrained keypoint configuration for a 640-input model, reporting 54.9 AP yolov5s6_pose_640_ti_lite_54p9_82p2_config.yaml yolov5s6_pose_640_ti_lite_54p9_82p2_config.yaml. That is a quantised TI-lite artifact, not the paper's float model at 960.

Specific differences:

• Keypoint decoding. The paper encodes keypoints relative to the anchor centre without normalisation §3.2. The YOLOX head decodes with 2*x - 0.5 + grid, scaled by the stride, which is a YOLOX-style anchor-free decode yolo_kpts_head.py.
• OKS denominator. The paper uses 2 s² k_n² in eq. 3 eq. 3. The code uses bounding-box area times 4 σ_n², where σ are COCO-style constants divided by 10 yolo_kpts_head.py [code: yolox/models/yolo_kpts_head.py#L771-L773]. If the paper's k_n² is matched to the code, that gives k_n² = 2σ_n². The paper does not state k_n, so this is inferred from code only.
• Keypoint visibility label. The paper uses δ(v_n > 0) eq. 4 §3. The code uses keypoint coordinate != 0 yolo_kpts_head.py. These match for ordinary labels but can differ at zero coordinates.
• Loss weights. The paper uses 0.5, 0.05, 0.1 and 0.5 for classification, box, keypoint and keypoint confidence eq. 5. The code uses 5.0 for box and keypoint, and 1.0 for objectness, classification and keypoint confidence [code: yolox/models/yolo_kpts_head.py#L511-L512]. The code also has an objectness term that eq. 5 does not contain.
• Augmentation. The paper uses scale [0.5, 1.5], translation ±10 and flip p = 0.5 §4.1. The YOLOX base config uses mosaic scale (0.1, 2), translation 0.1 as a fraction, degrees 10, shear 2.0, mixup, HSV colour and EMA [code: yolox/exp/yolox_base.py#L41-L63] yolox_base.py. The paper mentions none of mixup, shear or EMA.
• Optimiser. The paper gives base learning rate 1e-2 and cosine decay §4.1. The code uses 0.01/64 per image, so 1e-2 at batch 64, with warmup of 5 epochs, 15 no-augmentation epochs, weight decay 5e-4, momentum 0.9 with Nesterov, and a cosine schedule yolox_base.py [code: yolox/exp/yolox_base.py#L251-L276]. The paper gives none of these.
• Input size. The paper's tables use 960 tab. 1. The YOLOX human-pose experiment uses 640 for train and test yolox_base.py yolox_base.py. The shipped YOLOv5 artifact is 640 too.
• Pad value. The paper does not state the pad value for letterboxing §4.1. The shipped config pads with 114 yolov5s6_pose_640_ti_lite_54p9_82p2_config.yaml.
• Not found. I did not find the paper's YOLOv5-based keypoint head, OKS loss or training loop in either repo. The YOLOX repo is the only one with keypoint training code. The paper's own anchor-relative keypoint encoding §3.2 does not appear in the YOLOX code.

Pitfalls

• AP50 is the headline, AP is not. Abstract and §4.6 claim the lead on AP50. On AP, several bottom-up rows are higher, for example HigherHRNet W48 at 640 (69.9 on val2017 and 68.4 on test-dev2017) tab. 1 tab. 2. Compare on matching metrics.
• Compute is not matched. The paper says it achieves SOTA AP50 with about 4× less compute, citing YOLOv5m6-pose at 66.3 GMACS against DEKR at 283.0 GMACS on test-dev §1 tab. 2. Rows marked † have complexity adjusted for flip or multi-scale test, so compare them with other † rows only.
• Eq. 3 is hard to read. The sign inside the exponential is not legible in the copy we used. The standard OKS form has a negative exponent, which this card uses. The paper's line L = 1 − Σ OKS and the expanded ratio are reconciled here as one average over labelled keypoints.
• s is not defined precisely. Eq. 3 calls s the object scale. Whether this is area or its square root is not stated. The code uses box width times height yolo_kpts_head.py, which is area.
• k_n values are not stated. The paper says they come from COCO authors §3.4 but gives no numbers. The code's constants are a different scale (see Paper vs code).
• Classification loss is not defined. λ_cls = 0.5 appears in eq. 5, but the form of the classification loss is not given anywhere in the paper. Objectness is not in eq. 5 at all.
• Keypoint confidence label. The paper uses visibility greater than 0 as the label. The code uses coordinate non-zero. Implement the paper's flag if the dataset provides it.
• Keypoint loss weight depends on area. Dividing by 2 s² k² means small people get large similarity losses for the same pixel error. Check the object-area convention before training.
• Training details are missing. Batch size, momentum, weight decay, warmup, anchor shapes, the training input size and the colour augmentation values are not stated. Code values are a different model (YOLOX) and should not be copied without checking.
• Quantisation. 8-bit drops AP by about 6 points. Mixed precision needs about 30% of layers at 16-bit to recover most of that. Activation changed from SiLU to ReLU before quantisation, with a 1–2% drop §4.5. The text says mixed precision is 1.2% below float, and the table values give 1.3 (61.9 − 60.6). This is rounding in the paper.
• Post-training only. The paper says quantised models use post-training quantisation and not quantisation-aware training §4.5.
• ONNX export claim. The paper says the full model, including the per-person output, exports to ONNX §3.7. Whether box decoding and NMS are inside the exported graph is not stated.
• Keypoints outside the box. The model can predict keypoints outside the predicted box, which the authors present as an advantage §3.6, fig. 3.
• L1 baseline tuning. The authors say they had to tune the loss weights for L1, while the OKS loss was used as-is §4.3. Compare the ablation with that in mind.
• Table 4 and the resolution table. The resolution table in §4.4 has no caption or number. Cite it as §4.4.

Lineage

• Builds on: YOLOv5 [ref. 1], CSP-Darknet53 backbone [ref. 27] (CSPNet), PANet neck [ref. 6], CIoU and DIoU losses [ref. 37], and COCO keypoints [ref. 7]. The paper names none of these by arXiv id.
• Compared against (Tables 1 and 2): OpenPose [ref. 4], HigherHRNet [ref. 9], HRNet [ref. 16], DEKR [ref. 30], EfficientHRNet [ref. 15], Hourglass [ref. 12], PersonLab, PifPaf. Discussed without a table row: Associative Embedding [ref. 5], Mask R-CNN [ref. 20], SimpleBaseline [ref. 8].
• Superseded by: not stated.