Real-time 2D Multi-Person Pose Estimation on CPU: Lightweight OpenPose

In plain words

Pose estimation finds every person in a photo and marks their joints (ankles, knees, hips, elbows, wrists, shoulders, head points), so you get a stick-figure skeleton for each person. The OpenPose method does this well, but its original network is too heavy to run in real time on an ordinary computer processor without a graphics card.

This paper keeps OpenPose's approach and makes it lighter. The network reads a smaller image through a compact feature extractor, and it keeps only two prediction passes instead of several. Its layers are also redesigned to be cheaper while keeping their reach across the image. The post-processing code that joins joints into people was rewritten to avoid extra memory work. Together these cut the network's compute to about a seventh of the original's, and the author reports real-time speed on a mini PC and a laptop-class CPU, with accuracy close to the original.

TL;DR

• Lightweight OpenPose is a bottom-up multi-person 2D pose estimator for CPUs and edge devices §1.
• It uses a dilated MobileNet v1 backbone, two stages (one initial, one refinement), and residual lightweight refinement blocks in place of 7x7 convolutions §4.1.
• Author-reported: 9 GFLOPs and 42.8 AP on COCO validation against 61.7 GFLOPs and 43.4 AP for the 2-stage baseline tab. 2 tab. 5. End-to-end 28 fps on an Intel NUC and 26 fps on an i7-6850K tab. 6.
• Worth reading if you need pose estimation without a GPU. Do not trust the fps numbers as reproducible from the repository: its post-processing is in Python and the fps come from OpenVINO runs.

Problem

Bottom-up pose estimators (find all keypoints first, then group them into people) are robust to the number of people in the frame §2. The OpenPose baseline runs a VGG-19 backbone cut at conv4_2, two extra convolutions, an initial stage and five refinement stages §3.2. The 2-stage OpenPose network totals 61.7 GFLOPs tab. 2; the abstract's "about 15%" matches the 9 GFLOPs result against that total. The stated target is a network for edge devices, and real-time speed on a CPU without a graphics card [abstract] §1.

The paper's own evidence of the bottleneck: on a video with more than 20 poses the baseline ran at 1.17 fps on the NUC and 0.95 fps on the CPU tab. 6. Resizing feature maps to the input size took 10.5 fps in isolation and keypoint extraction 1.81 fps, while grouping itself was fast (454 fps) tab. 3.

Assumptions: the reported speed numbers apply to a video with more than 20 poses tab. 6; the accuracy numbers are COCO validation AP in single-scale testing §3.2 tab. 2.

Method

The paper has no numbered equations and no algorithm box. The method is three changes to OpenPose §4: a new backbone, a new refinement stage design, and a post-processing rewrite.

Pipeline. The image is scaled so its height matches the network input, with the width scaled to keep the aspect ratio, then padded to a multiple of 8 §3.1. The network outputs keypoint heatmaps and part affinity fields (PAFs) at 1/8 resolution §3.1. Keypoints are read from the heatmap peaks and grouped into people with the PAFs §3.1.

Step 1: backbone §4.1.1. The first-stage features come from MobileNet v1. The author removes the stride of the conv4_2 depthwise layer and sets dilation 2 on conv5_1, so the output stride stays at 8 and the receptive field is kept. The backbone runs through conv5_5 §4.1.1. Adding conv5_6 improved accuracy but cost speed §4.1.1. MobileNet v2 was tried and gave worse results §4.1.1 tab. 4.

Step 2: neck §4.1.2. Conv4_3 is replaced by three depthwise separable convolutions, with channels reduced from 256 to 128 §4.1.2 tab. 5. The paper does not describe conv4_4 separately. The code also adds a 1x1 projection and a 3x3 convolution after the residual sum with_mobilenet.py.

Step 3: initial stage §4.1.2 tab. 1 fig. 2. Heatmaps and PAFs share the trunk, which is three 3x3 convolutions with 128 channels. Each output has its own 1x1 convolution to 512 channels and a final 1x1 convolution to its output size. The two output branches are described in the paper as a single shared branch with the last two layers separate §4.1.2 fig. 2.

Step 4: refinement stage §4.1.2 fig. 3 tab. 5. The refinement input is the backbone features concatenated with the previous stage's heatmaps and PAFs §4.1.2 with_mobilenet.py. The same shared-trunk design is used. Each refinement block replaces a 7x7 convolution with a 1x1 convolution, a 3x3 convolution and a 3x3 convolution with dilation 2, plus a residual connection §4.1.2 fig. 3. The paper reports this block at about 2.5 times fewer operations than the 7x7 it replaces §4.1.2. The final network has one refinement stage §4.1 with_mobilenet.py.

Step 5: training of the final network §4.1. The network is trained with all refinement stages and then the first two are used §4.1. This gives about a 1% AP gain §4.1. The paper refers to "default training parameters from the original paper" and does not state them §4.1. Values used by the code are listed under Hyperparameters.

Step 6: post-processing §4.2 §3.1. The paper's output is upsampled by a factor of 8 instead of to the full input size, which keeps accuracy and removes the bottleneck §4.2. The paper uses factor 4 in its demo §4.2. The rest of the algorithm is in the code, not the paper.

# Lightweight OpenPose forward pass  [§4.1] [tab. 1] [fig. 2]
# Input:  image (B, 3, H, W), normalised as (pixel - 128) / 256
# Output: heatmaps and PAFs for every stage; the last two are used at inference

def forward(image):
    # 1. Dilated MobileNet v1 backbone up to conv5_5, output stride 8  [§4.1.1]
    #    [code: models/with_mobilenet.py#L92-L105]
    backbone = mobilenet_v1_dilated(image)           # (B, 512, H/8, W/8)

    # 2. Neck: depthwise separable blocks with residual, down to 128 ch  [§4.1.2]
    #    [code: models/with_mobilenet.py#L106]
    features = neck(backbone)                        # (B, 128, H/8, W/8)

    # 3. Initial stage: shared trunk, then 19 heatmaps and 38 PAFs  [§4.1.2]
    #    [code: models/with_mobilenet.py#L108]
    heatmaps, pafs = initial_stage(features)
    stages = [heatmaps, pafs]

    # 4. Refinement: backbone features plus last stage's estimates  [§4.1.2]
    #    [code: models/with_mobilenet.py#L118-L121]
    for stage in refinement_stages:
        inputs = concat(features, stages[-2], stages[-1])   # (B, 185, H/8, W/8)
        heatmaps, pafs = stage(inputs)
        stages += [heatmaps, pafs]

    return stages

# Lightweight refinement block, replaces a 7x7 convolution  [§4.1.2] [fig. 3]
# Input: x (B, C, H, W); output: (B, 128, H, W)

def refinement_block(x):
    # 1x1 convolution to 128 channels  [§4.1.2]
    # [code: models/with_mobilenet.py#L51]
    projected = conv1x1(x)                           # (B, 128, H, W)

    # 3x3 convolution, then 3x3 convolution with dilation 2  [§4.1.2]
    # [code: models/with_mobilenet.py#L53-L54]
    trunk = conv3x3(projected)
    trunk = conv3x3_dilated(trunk, dilation=2)

    # Residual connection  [§4.1.2]
    # [code: models/with_mobilenet.py#L60]
    return projected + trunk

# Keypoint extraction and grouping for inference  [§3.1] [§4.2]
# Factors and thresholds from code: demo uses upsample 4; paper uses 8 or 4 [§4.2]
# Input: heatmaps (H, W, 19), pafs (H, W, 38) at 1/8 resolution

def decode(heatmaps, pafs):
    # 1. Upsample the maps by factor 4 (demo) or 8 (paper's accuracy claim)
    #    [§4.2] [code: demo.py#L87]
    heatmaps = upsample(heatmaps, factor=4)
    pafs = upsample(pafs, factor=4)

    # 2. Keypoints are local maxima above 0.1; peaks within 6 px are merged
    #    [code: modules/keypoints.py#L11-L43]
    peaks = find_peaks(heatmaps, min_value=0.1, min_distance=6)

    # 3. For each of the 19 limb types, score every candidate pair by sampling
    #    10 points on the segment and projecting the PAF  [§3.1]
    #    [code: modules/keypoints.py#L64-L100]
    for each limb type in 19 limb types:
        score = mean of PAF projections above 0.05 along the segment
        keep candidate if score > 0 and more than 80% of samples pass
        keep at most one connection per keypoint, greedily by score
    # [code: modules/keypoints.py#L46-L61]

    # 4. Merge limbs into people; keep people with at least 3 keypoints
    #    and mean keypoint score >= 0.2  [code: modules/keypoints.py#L153-L158]
    return people

Figures are not reproduced; they are described and cited. Figure 1 is the OpenPose pipeline fig. 1. Figure 2 shows the single shared prediction branch for the initial stage fig. 2. Figure 3 shows the lightweight replacement for the 7x7 convolution fig. 3.

Hyperparameters

Paper values come first; code-only values are marked and cited to code only.

Results

Author-reported. Setup: COCO validation for AP, single-scale testing, 368 x 368 for the comparison §3.2; end-to-end fps on Intel NUC 6i7KYB (integrated GPU, FP16) and Core i7-6850K (FP32), OpenVINO R4 §4.3, on a video with more than 20 poses tab. 6.

Per-stage accuracy and cost of the proposed network tab. 5:

Backbone study, with original OpenPose initial and refinement stages tab. 4:

Pre-optimisation profile of post-processing, fps tab. 3: resize feature maps 10.5, extract keypoints 1.81, group keypoints 454, total 1.54.

Paper vs code

Code: official repository by the author [raw/code/1811_12004, commit d23c284b09acf27a163e1febd511e7482cac25ed]. Apache-2.0 [LICENSE]. The paper links only the OpenVINO Open Model Zoo and the OpenVINO toolkit for the model and C++ demo §4.3 §5. The OpenVINO path is not in this repository. The code's owner is the paper's sole author. The repository commit is dated 2024-04-30, later than the paper.

1. Accuracy. Paper: 42.8 AP for the 2-stage network tab. 5. README: the released checkpoint reaches about 40% AP on the full COCO validation set, and 38.6% on the validation subset, after 3-stage training [README.md#L3] [README.md#L45] [README.md#L96]. The 42.8 figure is not reproduced by the repository. The paper also says "COCO validation subset" §3.2 without naming the subset.
2. Input size. Paper: final input 456 x 256 §4.3. Code: the Python demo takes height 256 by default and derives width from the aspect ratio, then pads demo.py demo.py. The fixed 456 width is not in the code.
3. Upsampling and grouping. Paper: upsample by 8 to keep accuracy, 4 for the demo §4.2. Code: demo upsamples by 4 demo.py. Evaluation resizes heatmaps to the original image size, not by a factor val.py.
4. Post-processing speed. Paper: parallel keypoint extraction with OpenCV in C++ §4.2. Code: keypoint extraction and grouping are NumPy and Python keypoints.py keypoints.py. The README says the C++ demo, which is in OpenVINO, is the one for performance [README.md#L105-L109]. The fps in the paper cannot be reproduced from this repository.
5. Neck. Paper: conv4_3 replaced by three depthwise separable convolutions, 256 to 128 channels §4.1.2. Code: a 1x1 projection from 512 to 128, three depthwise separable blocks with ELU activations and no batch norm, a residual sum, then a 3x3 convolution with_mobilenet.py. The paper does not say how conv4_4 maps to the code.
6. Refinement block. Paper: 1x1, 3x3, 3x3 with dilation 2, residual §4.1.2. Code matches with_mobilenet.py. The code uses batch norm and ReLU, which the paper does not describe.
7. Loss. The paper states no loss. Code: masked L2 on heatmaps and PAFs at every stage, summed, scaled by 1/2 and by batch size loss.py train.py.
8. Training. The paper points to "default training parameters from the original paper" §4.1. The code uses Adam with the values in Hyperparameters, a MobileNet v1 initialisation from an external checkpoint, and a 3-step schedule [README.md#L40-L59] train.py. These are not in the paper.
9. Pose tracking and smoothing. Code adds a One Euro filter and pose tracking for video pose.py one_euro_filter.py. The paper does not describe them, and the reported fps do not account for them.

Pitfalls

• Accuracy claims are not reproducible from the repository as stated. The paper's 42.8 AP does not match the released checkpoint's roughly 40% AP [README.md#L3] tab. 5. Do not compare your numbers to 42.8 without the same validation subset and evaluation code.
• Speed is an OpenVINO C++ result. The fps in the paper come from OpenVINO with FP16 on the NUC and FP32 on the CPU §4.3 tab. 6. The Python demo is for preview only [README.md#L107-L110].
• Post-processing dominates after the network is shrunk. In the final numbers, network inference is about 33 fps while post-processing reaches 125 to 160 fps tab. 6. Before the optimisation, post-processing was the bottleneck tab. 3. Measure both parts.
• Upsampling factor is a speed-accuracy knob. Skipping the resize to input size lost accuracy, and factor 8 was claimed to match the full resize §4.2. The demo uses factor 4 demo.py. Test both on your data.
• Input normalisation and channel order. The model expects planar BGR, mean 128 and scale 1/256 [README.md#L95] val.py. The paper does not state the channel order or normalisation.
• Padding. Frames are scaled by height, padded to a multiple of 8, and cropped back after inference §3.1 val.py. Padding is centred. Output coordinates must be mapped back through the scale and padding demo.py.
• Grouping thresholds are not in the paper. Peak threshold 0.1, merge distance 6 px, PAF threshold 0.05, success ratio 0.8, minimum 3 keypoints and minimum mean score 0.2 are all from code only keypoints.py. They change results and speed; they are not tuned in the paper.
• Greedy grouping. Each limb type keeps at most one connection per keypoint, chosen greedily by score keypoints.py. Ambiguous crowds may group differently from a global optimisation.
• Backbone pretraining. The README starts from a MobileNet v1 checkpoint trained elsewhere [README.md#L47] train.py. The paper does not state its initialisation.
• Figure, table and section numbers follow caption and section order in the LaTeX source. The compiled PDF was not checked, so a number may differ by one in the PDF.
• The code is not the paper's network in every detail. Details such as ELU activations in the neck, and the batch norm placement in the refinement block, are code choices that the paper does not describe. Use the code as the reference for those layers.

Lineage

• Builds on: OpenPose [cao2017realtime], the bottom-up part-affinity-field method §1 §3. Changes: lighter backbone, refinement block and post-processing §4.
• Uses ideas from: MobileNet v1 [mobilenet_v1] and v2 [mobilenet_v2] (only v1 is used), dilated convolutions [drn], the CPM large-receptive-field idea [cpm], and residual connections [he2016resnet] §4.1.
• Compared against: the 2-stage OpenPose baseline tab. 2 tab. 6; Mask R-CNN keypoint demo, not measured §1 [maskrcnn2go]; a bottom-up method reported at 23 fps on a GTX 1080 Ti for 3 persons, not measured here §2 [kocabas18prn]; the open-source repository by ildoonet, not measured here §1 [ildoonet].
• Superseded by: not stated in this wiki.