Unsupervised keypoint detection

Overview

The task is to find a small set of points or regions on an object or scene without any annotated keypoints. The training signal comes from the data itself, and a downstream objective decides whether the points are useful. Four papers use the idea in two domains: three on image animation and one on 3D point-cloud registration.

In image animation, a network predicts keypoints from a frame, and the frame must be rebuildable from a source image plus those keypoints. A known image warp adds a second signal, because the keypoints should move with the warp 2003.00196 §3.1 2003.00196 §3.3 2203.14367 §3.3. The three image papers agree on this detector-plus-equivariance core and differ in how detections become motion. The first gives each point a local affine map through a 2x2 Jacobian 2003.00196 §3.1. The second groups its points and fits one thin-plate spline (TPS) warp per group 2203.14367 §3.1. The third drops the point for a region: each region's centre, extent and orientation come from the mean and covariance of its heatmap, and an SVD gives the principal axes 2104.11280 §3.2. All three use these detections only for image animation: a still image is moved by the motion recovered from a driving video 2003.00196 §1 2203.14367 §1 2104.11280 §3. Cards: First Order Motion Model for Image Animation, Thin-Plate Spline Motion Model for Image Animation and Motion Representations for Articulated Animation.

PRNet uses learned points for a different job. In each of two partial 3D point clouds it keeps the k points whose embeddings have the largest norm, matches them with a sampled assignment, and solves for the rigid motion in closed form 1910.12240 §3.2 1910.12240 eq. 9. Its loss has a registration term, a cycle term and a feature term, and no reconstruction or equivariance term 1910.12240 eq. 12 1910.12240 eq. 13 1910.12240 eq. 14 1910.12240 eq. 15. Card: PRNet: Self-Supervised Learning for Partial-to-Partial Registration.

Variants

Position-only keypoints

Each keypoint is a location, and its Jacobian is fixed to the identity. The paper says this recovers the zeroth-order motion of Monkey-Net, which it builds on 2003.00196 §3.1. In the paper's ablation this is the baseline row. That row also drops the pyramid loss and the occlusion map, so it does not isolate the effect of the Jacobian 2003.00196 tab. 1. Its Tai-Chi-HD values are L1 0.073, AKD 8.945, MKR 0.099 and AED 0.235 2003.00196 tab. 1. Card: First Order Motion Model for Image Animation.

Keypoints with local affine Jacobians

Each keypoint also carries a 2x2 Jacobian. The detector emits four extra channels per keypoint, which are averaged with the same heatmap weights 2003.00196 §3.1. Motion is then a first-order approximation around each keypoint 2003.00196 eq. 4 2003.00196 eq. 5. The Jacobian needs its own equivariance term to stay stable. Without that term, the Tai-Chi-HD scores are AKD 9.887 and MKR 0.052, against 6.862 and 0.036 for the full model 2003.00196 tab. 1. The motion representation is K × 6 = 60 floats with K = 10, compared with K × 5 = 50 for Monkey-Net and 128 for X2Face 2003.00196 §4. Card: First Order Motion Model for Image Animation.

Keypoint groups with thin-plate-spline warps

The detector outputs K groups of N = 5 points for each image, with K = 10 in the experiments 2203.14367 §3.1 2203.14367 §4.2. Each group of five source-driving pairs defines one TPS: the smoothest map through the pairs, found by minimising bending energy 2203.14367 eq. 1. Its closed form is an affine part plus radial terms in r² log r² 2203.14367 eq. 2 2203.14367 eq. 3. The background has its own affine map 2203.14367 eq. 4.

The K TPS warps and the background warp are combined per pixel by a softmax over K+1 contribution maps, and the flow is their weighted sum 2203.14367 eq. 5 2203.14367 eq. 6. During early training each TPS term is dropped at random, and the kept terms are rescaled 2203.14367 eq. 7 2203.14367 eq. 8. The motivation is that one affine piece per keypoint cannot represent curved motion, such as a lip curving open 2203.14367 §1. Occlusion masks are predicted at several decoder scales, and an hourglass inpainting network reads them through skip connections 2203.14367 §3.2 2203.14367 fig. 3. The paper does not state the keypoint network's architecture 2203.14367 §3.1. Card: Thin-Plate Spline Motion Model for Image Animation.

Heatmap moments for articulated parts

Each region is a softmax heatmap over pixels. Its centre is the heatmap-weighted mean of pixel coordinates, and its covariance around that centre is decomposed by SVD. The linear part of the region transform is U S^(1/2), with the centre as translation 2104.11280 eq. 5 2104.11280 eq. 6 2104.11280 eq. 7 2104.11280 §3.2. Shear is not represented, so each region has five degrees of freedom rather than six 2104.11280 §3.2. A region's motion is composed through a reference frame, and the dense flow is a per-pixel softmax mix of the K region motions and one background motion 2104.11280 eq. 3 2104.11280 eq. 8 2104.11280 §3.4. The background is a global 2x3 affine map from its own encoder 2104.11280 §3.3.

The paper argues that regressing local affine parameters needs higher-capacity networks and generalises poorly 2104.11280 §3.2 2104.11280 §4.1. Its support for this is a toy rotated-rectangles experiment, where the PCA head beats a regression head across training-set sizes from 32 to 1024. The authors say the gap exceeds an order of magnitude at the large-data end, and no table reports those numbers 2104.11280 §4.1 2104.11280 fig. 5.

Shape is separated from pose by an animation-via-disentanglement network: two encoders map the source regions (shape) and driving regions (pose) to 64-dimensional codes, and a decoder combines them 2104.11280 §3.6. The main model is trained end to end with a multi-scale VGG-19 reconstruction loss and an equivariance loss 2104.11280 §3.5 2104.11280 eq. 9 2104.11280 eq. 10. The shape-pose network is trained afterwards 2104.11280 §3.6. Card: Motion Representations for Articulated Animation.

Feature-norm points for point-cloud registration

This variant is not heatmap-based. The embeddings come from a DGCNN, which a Transformer then contextualises across both clouds 1910.12240 §3.2. The keypoints are the k points with the largest embedding norm, taken separately in each cloud 1910.12240 eq. 9. Keypoints from the two clouds are matched with a Gumbel-Softmax over feature scores, at a temperature predicted from the features. The gradient is a straight-through estimate 1910.12240 eq. 10 1910.12240 eq. 11. The rigid motion is the SVD closed form on the matched keypoints 1910.12240 eq. 2 1910.12240 eq. 8. Training runs three passes, moving the source after each, and the paper uses k = 512 points in training 1910.12240 §3.2 1910.12240 supp. Training Protocol. The paper says the keypoints are learned without supervision 1910.12240 §4.3. Its ModelNet40 partial split reports rotation MSE 10.235 against 45.005 for DCP-v2 [arxiv:1910.12240 tab. 1]. Card: PRNet: Self-Supervised Learning for Partial-to-Partial Registration.

When to use which

The first rows are for image animation. The registration row is last.

Implementation sketch

The sketch covers the detector, the equivariance term on positions, and relative transfer at test time, with the Jacobian head shown as an option. The TPS block shows where the image TPS variant differs. The region block shows the heatmap-moment variant. The last block is the point-cloud keypoint selection. Each step is labelled with its source in the paper.

# Keypoint detector, position-only or with local Jacobians  [arxiv:2003.00196 §3.1]
# Input:  frame (B, 3, H, W), resized internally to 64 x 64
# Output: positions (B, K, 2), Jacobians (B, K, 2, 2)

def detect(frame, use_jacobian):
    # One heatmap per keypoint, normalised over pixels  [arxiv:2003.00196 §3.1]
    heatmaps = softmax(keypoint_net(frame))   # (B, K, 64, 64)
    # Position is the expected grid coordinate under its heatmap  [§3.1]
    positions = sum(heatmaps * grid)          # (B, K, 2)
    if not use_jacobian:
        # Position-only variant: Jacobian fixed to identity  [§3.1]
        return positions, identity(B, K)
    # Four extra channels per keypoint, same weights as the heatmap  [§3.1]
    jac_maps = jacobian_net(frame)            # (B, K, 4, 64, 64)
    jacobians = sum(heatmaps * jac_maps)      # (B, K, 2, 2)
    return positions, jacobians

# Equivariance term on keypoint positions  [arxiv:2003.00196 §3.3]
# Input:  frame D, a sampled warp T (affine + thin-plate spline)
# Output: scalar loss

def position_equivariance(D, T):
    p, _ = detect(D, use_jacobian=False)
    p_warped, _ = detect(T(D), use_jacobian=False)
    # Warped detections, mapped back, should match the originals  [eq. 10] [eq. 11]
    return l1(p, T_points(p_warped))

# Relative transfer at test time  [arxiv:2003.00196 §3.4]
# Input:  source keypoints, driving keypoints, driving keypoints at frame 1

def transfer(kp_source, kp_driving, kp_driving_first):
    # Move each source keypoint by the driving motion since its first frame
    return kp_source + (kp_driving - kp_driving_first)

# TPS variant: one thin-plate spline per keypoint group  [arxiv:2203.14367 §3.1]
# Input:  K groups of 5 keypoints, driving and source, each (K, 5, 2)
# Output: K warp grids, plus one affine background grid

def fit_tps_fields(kp_driving, kp_source):
    fields = []
    for k in range(K):
        # Smoothest map through the five pairs  [eq. 1] [eq. 2] [eq. 3]
        # Direction of the fit: see Pitfalls
        fields.append(solve_tps(kp_driving[k], kp_source[k]))
    return fields

# Per-pixel weights over K TPS warps plus the background  [eq. 5] [eq. 6]
# Output: flow, the weighted sum of the K+1 warps

def combine(warps, logits):
    weights = softmax(logits, dim=parts)      # (K+1, H, W)
    return sum(weights[j] * warps[j] for j in range(K + 1))

# Region variant: centre and linear part from heatmap moments  [arxiv:2104.11280 §3.2]
# Input:  image (B, 3, h, w) at quarter resolution  [arxiv:2104.11280 app. C]
# Output: centres (B, K, 2), linear parts (B, K, 2, 2)

def measure_regions(image):
    # Softmax over pixels, one heatmap per region; no temperature in the paper
    heatmaps = softmax(region_net(image), dim=pixels)  # (B, K, h, w)
    # Centre is the heatmap-weighted mean of coordinates  [eq. 5]
    centres = sum(heatmaps * coords)                    # (B, K, 2)
    # Covariance of each heatmap around its centre  [eq. 6]
    offsets = coords - centres
    covars = sum(heatmaps * outer(offsets, offsets))    # (B, K, 2, 2)
    # Principal axes by SVD; linear part is U S^(1/2)  [eq. 6] [eq. 7]
    # The sign of each axis is not fixed by the paper; see Pitfalls
    U, S, _ = svd(covars)
    linear = U @ diag(sqrt(S))                          # (B, K, 2, 2)
    return centres, linear

# Point-cloud keypoints by feature norm, no keypoint labels  [arxiv:1910.12240 eq. 9]
# Input:  points (B, 3, N), per-point embeddings (B, d, N), k
# Output: keypoints (B, 3, k), keypoint embeddings (B, d, k)

def top_k_by_norm(points, feat, k):
    # Score each point by the L2 norm of its embedding  [eq. 9]
    score = norm(feat, dim=channels)          # (B, N)
    # Keep the k highest-scoring points  [eq. 9]
    idx = topk(score, k)                      # (B, k)
    return gather(points, idx), gather(feat, idx)

Contradictions

The three image-animation papers report Tai-Chi-HD numbers that cannot be compared directly. Each paper reports results for its own model, with its own baselines and training schedule. For the same dataset, the first paper reports its full model at L1 0.063, AKD 6.862 and MKR 0.036 2003.00196 tab. 1. The second reports MRAA at L1 0.048, AKD 5.41 and MKR 0.025, and its own model at L1 0.045, AKD 4.57 and MKR 0.018 2203.14367 tab. 1. The second paper says its metrics follow MRAA's evaluation setup 2203.14367 §4.2. The third reports its own model at L1 0.047, AKD 5.58 and MKR 0.027, and FOMM, retrained by its authors on a longer schedule, at L1 0.056, AKD 6.53 and MKR 0.033 2104.11280 tab. 2 2104.11280 §4.3. The third paper's FOMM row is lower than the first paper's own full model, which is the same method. The cards do not establish why; the schedule and evaluation setup are the likely differences, but neither is tested here. No wiki card runs the three methods head to head, so no result settles the others.

The third paper also conflicts in part with the first on regressed Jacobians. The first reports that its regressed local affine terms, with their own equivariance term, lower AKD and MKR on Tai-Chi-HD 2003.00196 tab. 1. The third argues that regressed affine parameters generalise poorly 2104.11280 §3.2. Its only isolated evidence is the toy experiment 2104.11280 §4.1. Its FOMM comparison changes the region representation and the background model together, so it does not test the head choice alone. Without its background model, the third paper's method has higher L1 than retrained FOMM (0.059 against 0.056) but lower AKD (5.55 against 6.53) 2104.11280 tab. 1.

PRNet does not conflict with the image papers. Its metric is rotation error on point clouds, and no metric is shared with them 1910.12240 tab. 1 2003.00196 tab. 3. Its numbers are not comparable to theirs.

Pitfalls

• K means different things. In the first image paper K = 10 is ten points, each with a Jacobian 2003.00196 §4. In the second, K = 10 is ten groups of five points, each group defining one TPS 2203.14367 §3.1. In the third, K = 10 is ten regions, each a heatmap with its own centre, extent and orientation 2104.11280 §3.2 2104.11280 tab. 2. Compare motion sizes by dimension, not by K. The first paper uses 60 floats for K = 10 2003.00196 §4.
• Motion dimension in the second paper. Its formula gives 134 for K = 8, while its text gives 118 2203.14367 §4.3. Check which holds before you quote the size of its comparison.
• Heatmap form (first paper). As printed, eq. 6 grows with distance, so it is not a bump-shaped heatmap. The released code uses a Gaussian 2003.00196 eq. 6 dense_motion.py.
• Softmax temperature (first paper). The paper does not state it. The shipped configs use 0.1 taichi-256.yaml. The third paper also states no temperature for its heatmap softmax 2104.11280 §3.2.
• Jacobian inversion (first paper). Relative and Jacobian transfer invert the driving Jacobian. The paper assumes local bijectivity, and the code has no explicit guard against near-singular matrices 2003.00196 §3.1 dense_motion.py.
• Relative or absolute transfer (first paper). The Fig. 1 caption says absolute transfer is used for Tai-Chi-HD, but the shipped config uses relative transfer 2003.00196 §3.4 taichi-256.yaml.
• Direction of the TPS constraint (second paper). The paper writes the constraint with the source keypoints mapped to the driving ones 2203.14367 eq. 1. The released code fits the other way and uses the result as the sampling grid util.py. Pick one direction and check it against the sampling step.
• TPS numerics (second paper). The released code adds a 0.01 ridge to the linear system and an epsilon inside the logarithm of the radial term. The paper gives exact interpolation and no epsilon 2203.14367 eq. 3 util.py.
• Dropout rescale (second paper). Eq. 7 divides the kept terms by 1 − P. The released code does not apply that division to the exponentials, so a literal reading and the code train differently. Pick one and say which 2203.14367 eq. 7 dense_motion.py.
• Equivariance input (second paper). Eq. 10 uses the source image, while the code applies the random warp to the driving frame 2203.14367 eq. 10 model.py.
• Loss scale (second paper). The text gives an unweighted sum of absolute values 2203.14367 eq. 14. The released code takes means and weights each term by 10 taichi-256.yaml.
• Hidden hyperparameters (second paper). The paper does not give the learning rate, optimiser, batch size, dropout probability, pyramid scales or the architecture of its three modules 2203.14367 §3. Those values must come from the code or be chosen by the implementer.
• Axis sign (third paper). The SVD returns each principal axis only up to sign, and the paper gives no sign rule for it 2104.11280 §3.2. An implementation has to choose one. The paper does not test what happens without one.
• Shape-pose scaling (third paper). Training randomly scales the driving regions per axis, which pushes shape to come from the source branch. The paper gives no range for the scale 2104.11280 §3.6.
• Table inconsistency (third paper). The Tai-Chi-HD 256 full-model L1 at K = 10 is 0.047 in tab. 1 and tab. 2, but 0.048 in tab. 4. Its AKD is 5.59 in tab. 1 and tab. 4 and 5.58 in tab. 2 2104.11280 tab. 1 2104.11280 tab. 2 2104.11280 tab. 4. Use the value from the table you cite.
• Region count is a tuning choice (third paper). From K = 5 to K = 20, the paper's own AKD on Tai-Chi-HD falls from 6.04 to 5.17, while retrained FOMM's AKD rises from 7.34 to 8.29 2104.11280 tab. 1. Choose K on validation data, not from these rows alone.
• Keypoint count is fixed in the image papers. All three image papers use K = 10 for their main results 2003.00196 §4 2203.14367 §4.2 2104.11280 tab. 2. The first paper links a baseline's failure on large pose changes to keypoints missing from the arms 2003.00196 §4.
• Keypoint count matters for registration. PRNet's ablation lowers accuracy sharply as k falls: MAE(R) is 27.843 at k = 16 and 1.454 at k = 512 1910.12240 supp. (d). The test-time k is not stated 1910.12240 §4.1.
• PRNet keypoints are not part-aligned. They are picked by feature norm, with no keypoint loss. The paper shows them in figures only, so do not read them as object parts 1910.12240 §4.3 1910.12240 fig. 4 1910.12240 fig. 5.
• PRNet matching is not one-to-one. The conclusion says the matchings are not bijective. A fix (Gumbel-Sinkhorn) is future work 1910.12240 §5 1910.12240 fig. 6.
• PRNet temperature has no stated range. The paper gives none for λ, so the predicted temperature is unbounded as written 1910.12240 §3.2. The released code clamps it model.py.
• PRNet test-time matching is random. The released code samples the Gumbel matching at inference as well model.py. The paper does not say whether it averages over samples or fixes a seed.
• PRNet gradient is biased. The straight-through estimator is biased, and the paper gives no variance or convergence comparison 1910.12240 §3.2.
• Evaluation depends on third-party detectors. AKD and MKR use an external landmark or pose detector in both image papers, so changing it changes the numbers 2003.00196 §4 2203.14367 §4.2.
• Human judgements are thin in the second paper. It uses 20 video pairs and does not report the number of raters or a significance test 2203.14367 §4.2 2203.14367 tab. 2. The first paper uses 50 input pairs with 10 raters per question 2003.00196 §4.
• TED-talks data licence (third paper). The source videos were shared under CC BY-NC-ND 4.0, which is non-commercial and no-derivatives, and the authors cut and rescaled them 2104.11280 app. E. The dataset is not in this wiki, and its numbers are author-reported. Check those terms before you redistribute or train on that data.
• Costs. The first image paper reports about 100k iterations at batch 20 to 40, roughly two days on two GPUs for Tai-Chi-HD and VoxCeleb 2003.00196 app. B. Each TPS fit solves an (N+3) × (N+3) linear system per group and per sample 2203.14367 eq. 2 2203.14367 eq. 3. The second paper reports about eight days of training on TED-talks 2203.14367 §4.2. PRNet reports about 10 hours on four V100 GPUs 1910.12240 supp. Training Protocol.

Papers

• First Order Motion Model for Image Animation: learns ten keypoints with local 2x2 Jacobians from video reconstruction and an equivariance loss, without object annotations.
• Thin-Plate Spline Motion Model for Image Animation: learns ten groups of five keypoints and warps with one thin-plate spline per group, plus multi-resolution occlusion masks.
• Motion Representations for Articulated Animation: measures ten heatmap regions by their mean and covariance (SVD), adds a global affine background and a shape-pose network, without part labels.
• PRNet: Self-Supervised Learning for Partial-to-Partial Registration: selects the top-k points by feature norm in two partial point clouds, matches them with Gumbel-Softmax and solves the rigid motion with SVD, without keypoint labels.