RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose

In plain words

Pose estimation finds the joints of people in a picture, such as the wrists, knees and head. The goal here is to do this fast enough for real-time use on modest hardware. The usual approach draws a heat map for each joint and picks the brightest spot. That works well but costs a lot of computation.

This paper builds a fast system in two steps. First a separate detector finds each person. Then a small network looks at the cropped person and finds each joint. The network does not draw heat maps. For each joint it guesses where the joint lies along the horizontal axis and along the vertical axis, by choosing among many small numbered slots on each axis. Because the output is two short lists of scores instead of large maps, the last part of the network stays small.

The authors did not invent one new trick. They tried many choices, one after another: the network's building blocks, the training recipe, the output format and the way the pieces are joined at run time. They kept the choices that helped and report how fast the result runs on a CPU, a graphics card and a mobile chip. The numbers are the authors' own.

TL;DR

RTMPose is a top-down pose estimator. A detector crops each person, and a CSPNeXt-based network predicts each keypoint as two 1-D SimCC distributions, one for x and one for y §3.1 §3.3. The head is a 7x7 convolution, a 256-dimensional expansion and one gated attention unit §3.3 §3.4. Training uses AdamW with flat-cosine decay, an EMA of the weights and a two-stage augmentation schedule §3.2 tab. 7. Author-reported: RTMPose-m reaches 75.8 AP on COCO with flip test, which the abstract gives without a table entry §1; RTMPose-s reaches 72.2 AP at 70+ FPS on a Snapdragon 865 chip §1.

Problem

Real-time multi-person pose estimation is hard on devices with little computing power. Earlier efficient networks and detection-free methods did not reach the accuracy that industrial use needs §1. Heat-map methods need high-resolution features or upsampling layers, which cost compute and latency §2 §3.1. The paper's premise is that a top-down pipeline can be fast if the detector is fast and the pose network is small §1.

Method

The pipeline has four parts. Each step below cites where the paper or the official code describes it.

1. Detection and crop. A real-time detector gives one box per person. The box is warped to the network input, 256x192 in the released configs §1 §3.5 rtmpose-m_8xb256-420e_coco-256x192.py rtmpose-m_8xb256-420e_coco-256x192.py. The paper's detectors are YOLOv3, Faster R-CNN, PicoDet-s and RTMDet §4.2 tab. 4.
2. Backbone. CSPNeXt-m, the backbone from RTMDet §3.1 rtmpose-m_8xb256-420e_coco-256x192.py.
3. SimCC head. A 7x7 convolution gives one map per keypoint §3.4 tab. 2. The maps are flattened, expanded to 256 dimensions and refined by one gated attention unit (GAU). Two linear layers output the x and y bin logits §3.3 rtmcc_head.py.
4. Training and decoding. Keypoints become Gaussian soft labels over bins. The loss is a KL divergence on temperature-softened distributions. At test time the argmax of each axis gives the coordinate §3.1 §3.4 simcc_label.py.

Pseudocode for the GAU, which is the only attention in the model §3.3 eq. 1 eq. 2:

# Gated Attention Unit over K keypoint tokens  [§3.3] [eq. 1]
# Input:  tokens (B, K, d), d = 256
# Output: tokens (B, K, d)
# Sizes:  e = 2 * d = 512, s = 128  [code: rtmcc_block.py#L176]

def gated_attention_unit(tokens):
    # Normalise the tokens before the block  [code: rtmcc_block.py#L197]
    normed = scale_norm(tokens)                       # (B, K, d)
    # One projection gives U, V and a shared base  [code: rtmcc_block.py#L187]
    # The activation is SiLU in the config; the paper leaves phi unstated
    hidden = activation(linear_uv(normed))            # (B, K, 2e + s)
    u, v, base = split(hidden, [e, e, s])
    # Q and K are per-dimension affine maps of the shared base  [code: #L254]
    queries, keys = unbind(base * gamma + beta)       # each (B, K, s)
    # Scaled scores, ReLU then square  [eq. 2] [code: rtmcc_block.py#L283]
    scores = queries @ transpose(keys)                # (B, K, K)
    attn = relu(scores / sqrt(s)) ** 2
    # Gate V-weighted context with U  [eq. 1] [code: rtmcc_block.py#L288]
    out = u * (attn @ v)                              # (B, K, e)
    out = linear_out(out)                             # (B, K, d)
    # Residual with a learnable scale, used when dims match [code: #L210]
    return out + res_scale * tokens

Pseudocode for the SimCC head, the training targets and the loss. The pseudocode blocks below follow the official code, which differs from the paper in places (see Paper vs code):

# SimCC head for a 256x192 crop  [§3.1] [§3.3] [§3.4] [tab. 2]
# Input:  feature map (B, C, 8, 6), height x width, from the backbone
# Output: logits_x (B, K, 384), logits_y (B, K, 512)  [code: #L5, #L204]

def simcc_head(feature_map):
    # 7x7 convolution, one map per keypoint  [§3.4] [tab. 2]
    maps = conv_7x7(feature_map)                      # (B, K, 8, 6)
    # Flatten, normalise, then expand to 256 dims  [§3.3] [code: #L111-L113]
    tokens = linear_256(scale_norm(flatten(maps)))    # (B, K, 256)
    tokens = gated_attention_unit(tokens)             # [§3.3] [eq. 1]
    # Two linear layers give the bin logits  [§3.1] [code: #L131-L132]
    logits_x = linear_x(tokens)                       # (B, K, 2 * 192)
    logits_y = linear_y(tokens)                       # (B, K, 2 * 256)
    return logits_x, logits_y

# Gaussian SimCC target for one visible keypoint  [§3.4] [eq. 4] [eq. 6]
# Input:  keypoint (x, y) in crop pixels; sigma_x, sigma_y (see eq. 6)
# Output: target_x (384,), target_y (512,)

def simcc_target(keypoint, sigma_x, sigma_y):
    # Map pixels to bins, split ratio 2  [code: simcc_label.py#L185]
    mu_x, mu_y = round(keypoint * 2)
    bins_x = arange(384)
    bins_y = arange(512)
    # Unnormalised Gaussian over the bins  [code: simcc_label.py#L278-L279]
    target_x = exp(-(bins_x - mu_x) ** 2 / (2 * sigma_x ** 2))
    target_y = exp(-(bins_y - mu_y) ** 2 / (2 * sigma_y ** 2))
    return target_x, target_y

# Temperature-softened KL loss on SimCC outputs  [§3.4] [eq. 5]
# Input:  logits and targets per axis (B, K, W), visibility weights (B, K)
# Output: scalar loss

def simcc_loss(logits_x, logits_y, targets_x, targets_y, weights):
    # beta = 1 / tau, with tau = 0.1 [§3.4] [code: classification_loss.py#L147]
    beta = 10.0
    total = 0
    for logits, targets in [(logits_x, targets_x), (logits_y, targets_y)]:
        # Softmax over bins, applied to model output and to the label
        log_probs = log_softmax(logits * beta)        # over bins
        label_probs = softmax(targets * beta)         # over bins
        # KL per keypoint, averaged over bins  [code: #L149-L150]
        kl = mean(label_probs * (log(label_probs) - log_probs), bins)
        # Weighted, summed over the batch and both axes  [code: #L175]
        total = total + sum(kl * weights)
    # Divide by the number of keypoints K  [code: classification_loss.py#L177]
    return total / K

# Decoding, with the flip test  [code: rtmcc_head.py#L196-L207]
# Input:  logits_x (K, 384), logits_y (K, 512), optional flipped pair
# Output: keypoint (x, y) in crop pixels, and a score per keypoint

def decode(logits_x, logits_y, flipped=None):
    if flipped is not None:
        # Average raw logits of the original and the mirrored pass
        logits_x = (logits_x + flip_back(flipped.x)) * 0.5
        logits_y = (logits_y + flip_back(flipped.y)) * 0.5
    # Argmax per axis, then undo the split ratio 2  [code: simcc_label.py#L173]
    x = argmax(logits_x) / 2
    y = argmax(logits_y) / 2
    # Score: smaller of the two peaks  [code: post_processing.py#L85-L86]
    score = min(max(logits_x), max(logits_y))
    return (x, y), score

Inference, as the paper describes it §3.5: the detector runs only every K frames (skip-frame, as in BlazePose), and in between the boxes come from the last pose result. The poses are then filtered with OKS-based pose NMS and a OneEuro filter §3.5. I did not find the skip-frame logic or the OneEuro filter in the code I read (see Pitfalls).

Hyperparameters

Values for RTMPose-m (COCO, 256x192). The paper's table covers all sizes tab. 7. The config is the fine-tuning config, and the pre-training config is listed in the last rows.

Results

Author-reported. Training data: COCO train2017 (118K images) and val2017 (5K) §4.2. Hardware: eight NVIDIA A100 GPUs §4.1. Metric: mean AP §4.1.

Development chain on COCO val (§3, author-reported). The paper gives a step-by-step gain from a lightweight SimCC baseline.

Table 1 (computational cost of the baselines) is a separate comparison: SimCC* at 4.03 GFLOPs and 23.59M parameters reaches 71.3 AP, and the caption does not state the backbone tab. 1.

COCO val2017, top-down with a given detector (flip test not used; GFLOPs of the pose model only) tab. 4. The RTMPose rows are complete. Baseline rows are in the paper and are not repeated here.

The abstract and the README give 75.8 AP for RTMPose-m with flip test. That number is not in any table §1 tab. 4.

COCO-SinglePerson, 1045 images from val2017 (flip not used; GFLOPs include detector and pose) tab. 5. Setup: RTMDet-nano at 320x320 detector, AIC (220K) tab. 5.

(*) Double inference, per the paper's table note tab. 5.

COCO-WholeBody v1.0, flip test used tab. 6. Whole-body AP and AR.

(*) Pre-trained on AIC+COCO tab. 6.

Other datasets, flip test used tab. 8 tab. 9.

Baselines in the same tables (HRNet, SimpleBaseline, SimCC, TokenPose) are between 63.7 and 90.2 tab. 8 tab. 9.

Speed, batch 1 tab. 10 tab. 11 tab. 12 §4.3. Setup: ncnn on a Snapdragon 865 with 4 threads, ONNX Runtime on an Intel i7-11700 with one thread, and TensorRT FP16 on a GTX 1660 Ti. Each run had 50 warm-up and 200 timed iterations §4.3. The paper says these were run with MMDeploy §4.3.

Pipeline speed with RTMDet-nano (320x320) as the detector tab. 12:

Paper vs code

Code: the official MMPose 1.x branch, linked from the paper, at commit db4ebd6e0cf1b113e20a7cbf94cdaba4ca2fcb85. The paper links open-mmlab/mmpose/tree/1.x/projects/rtmpose in the abstract. The repository is Apache-2.0 LICENSE.

• Attention normalisation. Paper: A = (1/n) relu^2(...), and n is not defined in the text §3.3 eq. 2. Code: the same ReLU-squared kernel with no 1/n factor rtmcc_block.py.
• GAU projections. Paper: separate U and V projections from X, with Z feeding the attention §3.3 eq. 1 eq. 2. Code: one linear layer produces U, V and a shared base, which is then turned into Q and K by per-dimension scale and offset, after a scale norm rtmcc_block.py [code: rtmcc_block.py#L197] rtmcc_block.py.
• GAU activation. Paper: phi is not specified §3.3. Code: SiLU in the config rtmpose-m_8xb256-420e_coco-256x192.py and ReLU as the class default rtmcc_head.py.
• Batch size. Paper: 1024 tab. 7. Config: 256 per data loader on eight GPUs (the paper's hardware), which is 2048 in total [code: projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py#L11]. The auto-scale reference batch is 1024 . The paper does not say which setting produced the reported numbers.
• Two-stage split. Paper: §3.2 gives 180 strong epochs and 30 weak epochs, and tab. 7 lists 210 epochs for pre-training §3.2 tab. 7. The paper gives no split for the 420-epoch fine-tuning. The config uses 390 strong and 30 weak for it .
• Channel attention in the backbone. Config enables it [code: projects/rtmpose/rtmpose/body_2d_keypoint/rtmpose-m_8xb256-420e_coco-256x192.py#L71]. The paper does not mention it §3.1.
• Sigma constants. Paper gives sigma = sqrt(W_S / 16) eq. 6. The config uses 4.9 and 5.66, which match 384 and 512 bins. Other input sizes must recompute the formula .
• Flip test. Paper: method not stated; tables mark some results as flip and some as no-flip tab. 4 tab. 6. Code: averages the raw logits of the two passes before argmax rtmcc_head.py.
• Speed headline. Abstract: 430+ FPS on a GTX 1660 Ti for RTMPose-m §1. Table 11 gives 2.288 ms for the pose model alone, with TensorRT FP16, batch 1 tab. 11. The pipeline with RTMDet-nano and RTMPose-m takes 4.312 ms on the GPU tab. 12.
• Backbone initialisation. The RTMPose-m config starts from a UDP checkpoint trained on AIC+COCO . The COCO-only rows in tab. 4 are a separate setting §4.2 tab. 4.

Pitfalls

• The headline 75.8 AP has no table row. It is stated in the abstract and the intro with flip test §1. The table row for RTMPose-m with the best detector is 75.7 (RTMDet-m detector, flip off) tab. 4. Do not take 75.8 as a reproduction target without the detector and flip setting.
• RTMPose-s AP differs by table. The abstract's 72.2 AP is a COCO number §1. Table 10 gives AP(GT) 72.8 for the same model, and the paper does not define the GT column tab. 10.
• Speed numbers are pose-only unless noted. The 430+ FPS figure is the pose model alone, batch 1, TensorRT FP16 tab. 11. A detector plus pose pipeline runs at 4.312 ms per frame on the GPU, about 232 FPS by 1000 / 4.312 (my arithmetic) tab. 12.
• Table 1 disagrees with the text. The text gives 69.7 AP for the CSPNeXt-m baseline §3.1, while Table 1 lists SimCC* at 71.3 tab. 1. The caption does not name the backbone for Table 1.
• Loss scale depends on batch size. The code sums over the batch and averages over bins, then divides by K classification_loss.py. The loss grows with batch size, so a learning rate tuned for one batch size does not transfer.
• Weak-stage and Cutout details are in the code only. The paper gives "small" rotation and "turn off shift" with no numbers §3.2. The config uses scale [0.75, 1.25] and rotation 60 . The CoarseDropout hole sizes are in code only .
• Split ratio is config-only. The bin count W_S in eq. 6 depends on a split ratio of 2 bins per pixel. The paper never states that ratio; the config sets it .
• Skip-frame, pose NMS and OneEuro are not in the code I read. The paper describes them §3.5. The repo's project README mentions the pipeline, but I did not locate the implementation. Treat that part of the pipeline as unverified.
• Input normalisation constants are code-only . An implementation that uses ImageNet statistics will not match the released weights.
• Decoding is argmax with no sub-bin refinement. DARK is off in the config . The paper's HRNet+DARK baseline is a separate model tab. 11.
• Learning-rate warm-up and gradient clipping are code-only .
• The paper does not describe decoding. The code takes the argmax of each axis and divides by the split ratio, and uses the smaller peak as the score simcc_label.py post_processing.py.

Lineage

• Builds on: RTMDet for the CSPNeXt backbone and the training recipe §3.1 §3.2.
• Builds on: SimCC, the coordinate-classification formulation §3.1. The paper's bibliography entry has no arXiv id, and no card exists for it.
• Builds on: the GAU attention block from "Transformer Quality in Linear Time" §3.3. The bibliography has no arXiv id, and no card exists for it.
• Builds on: SORD ordinal-regression soft labels §3.4. No card exists for it.
• Compared against: heat-map methods (HRNet, SimpleBaseline), TinyPose, MoveNet, BlazePose and ZoomNAS tab. 4 tab. 5 tab. 6 tab. 10.
• Superseded by: not known from the content of this wiki.