--- kind: concept slug: multi-person-pose-estimation title: Multi-person pose estimation summary: >- Find each person in an image and their body keypoints, each keypoint assigned to its person. Families: bottom-up grouping, top-down detect-then-estimate, and single-stage keypoints on a detector head. Cards cover one bottom-up method built for CPUs and one single-stage method. Latency, compute, precision and the reported metric decide the choice. tags: [computer-vision, pose-estimation, keypoints, object-detection] papers: ["1811.12004", "2204.06806"] related: [object-detection, instance-segmentation, post-training-quantization, quantization-aware-training, label-assignment, unsupervised-keypoint-detection, depthwise-separable-convolution, efficient-vision-backbone] status: draft updated_at: 2026-11-01 --- ## Overview The output is a set of people. Each person has a box, a score and a list of body keypoints such as wrists, knees and nose, and each keypoint must belong to exactly one person. The benchmark used in both source cards is COCO keypoints. The YOLO-Pose card scores with Object Keypoint Similarity (OKS) and reports AP [arxiv:2204.06806 §4.1]; the Lightweight OpenPose card reports AP on COCO validation [arxiv:1811.12004 §3.2]. The invariant all variants keep is that the keypoints of each person are attached to that person. The families differ in where that attachment happens. Bottom-up methods find keypoints across the whole image and group them after the fact. Top-down methods detect people first and estimate a pose inside each box. Single-stage methods predict the pose at the same time as the box, so the grouping is built into the output [arxiv:2204.06806 §1, §2.1, §2.2] [arxiv:1811.12004 §1, §2]. ## Variants ### Bottom-up keypoints with grouping (lightweight OpenPose) Bottom-up methods predict keypoints and limb fields for the whole image in one pass, then assemble people from them. Their cost does not grow with the number of people, which the source card gives as their main advantage [arxiv:1811.12004 §2]. The grouping step is the expensive part on a CPU: in the paper's profile of the baseline, grouping ran fast, but resizing feature maps and extracting keypoints did not [arxiv:1811.12004 tab. 3]. Lightweight OpenPose keeps the OpenPose approach and makes each part lighter [arxiv:1811.12004 §4]: - A dilated MobileNet v1 backbone through conv5_5, with output stride 8 [arxiv:1811.12004 §4.1.1]. - A neck where three depthwise separable convolutions reduce the channels from 256 to 128 [arxiv:1811.12004 §4.1.2] [tab. 5]. - Refinement blocks made of a 1x1 convolution, a 3x3 convolution and a dilated 3x3 convolution, with a residual connection. The paper reports about 2.5 times fewer operations than the 7x7 convolution they replace [arxiv:1811.12004 §4.1.2]. - Faster post-processing, where the upsampling factor trades accuracy against speed [arxiv:1811.12004 §4.2]. Author-reported: 9 GFLOPs and 42.8 AP on COCO validation, against 61.7 GFLOPs and 43.4 AP for the 2-stage OpenPose baseline [arxiv:1811.12004 tab. 2] [tab. 5]. The paper does not label the 43.4 row as the baseline; the card matches it by GFLOPs. End-to-end speed is 28 fps on an Intel NUC and 26 fps on an i7-6850K, measured on a video with more than 20 poses with OpenVINO [arxiv:1811.12004 tab. 6] [§4.3]. Card: [Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose](/papers/1811.12004). Its training values are not stated in the paper, which points to the original OpenPose training parameters [arxiv:1811.12004 §4.1]. ### Single-stage keypoints on a detector head (YOLO-Pose) A YOLOv5-style detector gets a second output per anchor. Each anchor predicts a box, a class score, and 17 COCO keypoints with a confidence for each. The keypoints of a person are stored with the anchor that matched that person, so no grouping step is needed [arxiv:2204.06806 §3, §3.2]. The keypoint regression is trained with a loss built from OKS, which the authors chose so that training targets the evaluation metric [arxiv:2204.06806 §3.4]. Card: [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](/papers/2204.06806). Reported result: 90.2 AP50 on COCO val2017 and 90.3 AP50 on test-dev2017 for the largest model, with no test-time augmentation. AP on the same rows is 69.4 and 68.5 [arxiv:2204.06806 tab. 1, tab. 2]. Model sizes run from YOLOv5s6-pose (15.1M parameters, 22.8 GMACS at 960 input) to YOLOv5l6-pose (87.0M, 145.6 GMACS) [arxiv:2204.06806 tab. 1]. ### Top-down detect-then-estimate Named as background only. The source card says these methods run a person detector and then a pose network per person, so their cost grows with the number of people [arxiv:2204.06806 §2.1]. No card here measures them beyond the authors' comparison rows in [arxiv:2204.06806 tab. 1, tab. 2]. ## When to use which The source cards are the only evidence on this page, so the table only claims what they report. The authors' own results are not independently checked. | Situation | Use | Why | |---|---|---| | You must run in real time on a CPU with no GPU | Lightweight OpenPose | Author-reported 26 to 28 fps at 9 GFLOPs, using OpenVINO [arxiv:1811.12004 tab. 5] [tab. 6]. Those fps are not from the Python code path [arxiv:1811.12004 §4.3] | | You need the people and their keypoints from one forward pass, with no grouping code | Single-stage keypoints on a detector head | Keypoints are stored per person by construction, and the output uses standard detection post-processing [arxiv:2204.06806 §1, §3.2, §3.7] | | Crowd size varies a lot and per-person cost must stay fixed | Bottom-up or single-stage | Top-down cost grows with each person [arxiv:2204.06806 §2.1]; bottom-up cost is set by the image [arxiv:1811.12004 §2] | | You need the highest AP50 on COCO val2017 or test-dev2017 | Check the rows in [arxiv:2204.06806 tab. 1] and [tab. 2] at matching input size before choosing | The largest single-stage model has the top AP50 but not the top AP among the listed rows [arxiv:2204.06806 tab. 1] | | Compute budget is small, around 20 to 25 GMACS | YOLOv5s6-pose at 960 input | 22.8 GMACS, 63.8 AP on val2017 [arxiv:2204.06806 tab. 1] | | Compute budget is under 30 GMACS and the input must be small | Check the low-complexity table | At 384 input, YOLOv5-s6 is reported at 3.65 GMACS and 44.9 AP, against 35.7 AP for EfficientHRNet-H4 at 4.2 GMACS [arxiv:2204.06806 tab. 4]. Those are the authors' numbers from different training setups | | Target is an int8 accelerator | Keep 16-bit or mixed precision | Post-training 8-bit drops AP from 61.9 to 55.6 on the authors' model. Mixed precision, with about 30% of layers at 16-bit, reaches 60.6 [arxiv:2204.06806 tab. 5] | | Training must be reproduced from the card alone | Expect extra work | Several training values are not stated: for YOLO-Pose, batch size, momentum, weight decay and the OKS per-keypoint weights [arxiv:2204.06806 §3.4, §4.1]; for Lightweight OpenPose, the optimiser and schedule [arxiv:1811.12004 §4.1] | ## Implementation sketch Two cores are shown: bottom-up grouping from limb fields, and the single-stage inference and loss. Each step carries its paper citation. ```pseudocode # Bottom-up grouping from keypoint maps and limb fields [arxiv:1811.12004 §3.1] # Input: heatmaps (H, W, K), limb fields (H, W, 2L) at 1/8 input scale # Output: people, each a set of keypoints def group_people(heatmaps, limb_fields, up): # Upsample before reading peaks; the factor trades speed for accuracy [§4.2] heatmaps = upsample(heatmaps, up) limb_fields = upsample(limb_fields, up) # Candidate keypoints are local peaks of the heatmaps [§3.1] peaks = find_peaks(heatmaps) connections = [] for limb in limb_types: # Score each candidate pair by how well the field points along the limb [§3.1] # Sample count and cut-offs are not stated in the paper connections += score_pairs(peaks, limb_fields, limb) # Merge connections that share a keypoint into people [§3.1] return merge_into_people(connections) ``` ```pseudocode # Single-stage pose inference [arxiv:2204.06806 §3] [eq. 1] [fig. 2] # Input: image (3, H, W) # Output: per person a box, a class score and 17 keypoints (x, y, conf) def predict_people(image): # Backbone and neck give feature maps at several scales [§3.1] [fig. 2] feature_maps = neck(backbone(image)) # Each anchor predicts 6 box values and 51 keypoint values [eq. 1] raw = [head(fmap) for fmap in feature_maps] # Boxes go through standard object-detection post-processing [§1] people = detection_postprocess(raw) for person in people: # Keep keypoints whose confidence is above 0.5 [§3] person.keypoints = [ kpt for kpt in person.keypoints if kpt.conf > 0.5 ] return people ``` ```pseudocode # Training loss for one matched anchor [eq. 2] [eq. 3] [eq. 4] [eq. 5] # Summed over all matched scales, anchors and locations [eq. 5] def pose_loss(pred, target): # Box term: 1 minus CIoU [eq. 2] box_loss = 1 - ciou(pred.box, target.box) # Keypoint term: 1 minus OKS over labelled keypoints [eq. 3] [§3.4] kpt_loss = 1 - oks(pred.kpts, target.kpts, target.labelled) # Keypoint confidence: binary cross-entropy against visibility [eq. 4] kpt_conf_loss = bce(pred.kpt_conf, target.labelled) # Classification term; its form is not stated in the paper [eq. 5] cls_loss = classification_loss(pred.cls, target.cls) # Weights 0.5, 0.05, 0.1 and 0.5 [eq. 5] [§3.4] return ( 0.5 * cls_loss + 0.05 * box_loss + 0.1 * kpt_loss + 0.5 * kpt_conf_loss ) ``` Where variants differ: the decoding of keypoints, the OKS scale and the per-keypoint weights. The single-stage paper encodes keypoints relative to the anchor centre without normalisation [arxiv:2204.06806 §3.2], and its per-keypoint constants `k_n` are not given [arxiv:2204.06806 §3.4]. The bottom-up paper gives its grouping thresholds only in code, not in the text [arxiv:1811.12004 §3.1]. Check the cards' pitfalls before porting. ## Contradictions - **Headline metric.** The YOLO-Pose abstract and discussion lead with AP50. On AP, several bottom-up rows are higher, for example HigherHRNet W48 at 640 with 69.9 on val2017 [arxiv:2204.06806 tab. 1]. Both numbers are reported by the authors, so compare on matching metrics. - **Compute units and matching.** The YOLO-Pose paper reports GMACS and the Lightweight OpenPose paper reports GFLOPs [arxiv:2204.06806 tab. 1] [arxiv:1811.12004 tab. 2]. Convert before comparing. The YOLO-Pose authors claim about 4× less compute for high AP50, citing YOLOv5m6-pose against DEKR W48 on test-dev [arxiv:2204.06806 §1] [tab. 2]. Their tables mark flip and multi-scale test rows with multiplied complexity, and those rows are only comparable with each other [arxiv:2204.06806 tab. 1]. - **Accuracy across papers.** Lightweight OpenPose reports 42.8 AP on a COCO validation subset at a final input of 456 x 256 [arxiv:1811.12004 §3.2] [§4.3]. YOLO-Pose reports 63.8 AP on val2017 at 960 input [arxiv:2204.06806 tab. 1]. The subsets, inputs and models differ, so the two numbers are not comparable. - **Reproduced accuracy.** The Lightweight OpenPose card records that the released checkpoint reports about 40% AP, not the paper's 42.8. See the Paper vs code section of that card. - **Paper and code.** The YOLO-Pose paper's keypoint head is on YOLOv5. The code that holds keypoint training is in a YOLOX repository, with a different decode, a different loss weighting and a 640 input, not the 960 input of the tables [arxiv:2204.06806 §3.1] [fig. 2]. This page does not treat the code as the paper's method. - **Loss baseline.** The YOLO-Pose authors say the L1 baseline needed loss-weight tuning and OKS did not [arxiv:2204.06806 §4.3]. The ablation in [arxiv:2204.06806 tab. 3] may therefore favour the OKS row. ## Papers - [Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose](/papers/1811.12004): bottom-up keypoints and limb fields with a lighter backbone, neck and refinement, for CPU real time. - [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](/papers/2204.06806): single-stage keypoints on a detector head, trained with an OKS-based keypoint loss.