---
kind: concept
slug: unsupervised-keypoint-detection
title: Unsupervised keypoint detection
summary: >-
  Learns a few points or part regions per input without labels, trained through a downstream objective. Image animation: keypoints with local 2x2 Jacobians, thin-plate-spline warps over groups of five, or heatmap-moment regions for articulated bodies. Point clouds: the top-k points by feature norm, for registration. Author-reported numbers are not comparable across papers.
tags: [keypoints, self-supervised, motion-transfer, equivariance, image-animation, articulated-objects, point-cloud, registration]
papers: ["1910.12240", "2003.00196", "2104.11280", "2203.14367"]
related: [equivariance-constraint, self-supervised-learning, disentangled-representation, multi-scale-feature-fusion, thin-plate-spline, iterative-closest-point, local-feature-matching]
status: draft
updated_at: 2026-11-01
---

## Overview

The task is to find a small set of points or regions on an object or scene
without any annotated keypoints. The training signal comes from the data
itself, and a downstream objective decides whether the points are useful. Four
papers use the idea in two domains: three on image animation and one on 3D
point-cloud registration.

In image animation, a network predicts keypoints from a frame, and the frame
must be rebuildable from a source image plus those keypoints. A known image
warp adds a second signal, because the keypoints should move with the warp
[arxiv:2003.00196 §3.1] [arxiv:2003.00196 §3.3] [arxiv:2203.14367 §3.3]. The
three image papers agree on this detector-plus-equivariance core and differ in
how detections become motion. The first gives each point a local affine map
through a 2x2 Jacobian [arxiv:2003.00196 §3.1]. The second groups its points
and fits one thin-plate spline (TPS) warp per group [arxiv:2203.14367 §3.1].
The third drops the point for a region: each region's centre, extent and
orientation come from the mean and covariance of its heatmap, and an SVD
gives the principal axes [arxiv:2104.11280 §3.2]. All three use these
detections only for image animation: a still image is moved by the motion
recovered from a driving video [arxiv:2003.00196 §1]
[arxiv:2203.14367 §1] [arxiv:2104.11280 §3]. Cards:
[First Order Motion Model for Image Animation](/papers/2003.00196),
[Thin-Plate Spline Motion Model for Image Animation](/papers/2203.14367) and
[Motion Representations for Articulated Animation](/papers/2104.11280).

PRNet uses learned points for a different job. In each of two partial 3D
point clouds it keeps the k points whose embeddings have the largest norm,
matches them with a sampled assignment, and solves for the rigid motion in
closed form [arxiv:1910.12240 §3.2] [arxiv:1910.12240 eq. 9]. Its loss has a
registration term, a cycle term and a feature term, and no reconstruction or
equivariance term [arxiv:1910.12240 eq. 12] [arxiv:1910.12240 eq. 13]
[arxiv:1910.12240 eq. 14] [arxiv:1910.12240 eq. 15]. Card:
[PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240).

## Variants

### Position-only keypoints

Each keypoint is a location, and its Jacobian is fixed to the identity. The
paper says this recovers the zeroth-order motion of Monkey-Net, which it
builds on [arxiv:2003.00196 §3.1]. In the paper's ablation this is the
baseline row. That row also drops the pyramid loss and the occlusion map, so
it does not isolate the effect of the Jacobian [arxiv:2003.00196 tab. 1].
Its Tai-Chi-HD values are L1 0.073, AKD 8.945, MKR 0.099 and AED 0.235
[arxiv:2003.00196 tab. 1]. Card: [First Order Motion Model for Image Animation](/papers/2003.00196).

### Keypoints with local affine Jacobians

Each keypoint also carries a 2x2 Jacobian. The detector emits four extra
channels per keypoint, which are averaged with the same heatmap weights
[arxiv:2003.00196 §3.1]. Motion is then a first-order approximation around
each keypoint [arxiv:2003.00196 eq. 4] [arxiv:2003.00196 eq. 5]. The Jacobian
needs its own equivariance term to stay stable. Without that term, the
Tai-Chi-HD scores are AKD 9.887 and MKR 0.052, against 6.862 and 0.036 for the
full model [arxiv:2003.00196 tab. 1]. The motion representation is K × 6 = 60
floats with K = 10, compared with K × 5 = 50 for Monkey-Net and 128 for X2Face
[arxiv:2003.00196 §4]. Card: [First Order Motion Model for Image Animation](/papers/2003.00196).

### Keypoint groups with thin-plate-spline warps

The detector outputs K groups of N = 5 points for each image, with K = 10 in
the experiments [arxiv:2203.14367 §3.1] [arxiv:2203.14367 §4.2]. Each group
of five source-driving pairs defines one TPS: the smoothest map through the
pairs, found by minimising bending energy [arxiv:2203.14367 eq. 1]. Its
closed form is an affine part plus radial terms in r² log r²
[arxiv:2203.14367 eq. 2] [arxiv:2203.14367 eq. 3]. The background has its own
affine map [arxiv:2203.14367 eq. 4].

The K TPS warps and the background warp are combined per pixel by a softmax
over K+1 contribution maps, and the flow is their weighted sum
[arxiv:2203.14367 eq. 5] [arxiv:2203.14367 eq. 6]. During early training each
TPS term is dropped at random, and the kept terms are rescaled
[arxiv:2203.14367 eq. 7] [arxiv:2203.14367 eq. 8]. The motivation is that one
affine piece per keypoint cannot represent curved motion, such as a lip
curving open [arxiv:2203.14367 §1]. Occlusion masks are predicted at several
decoder scales, and an hourglass inpainting network reads them through skip
connections [arxiv:2203.14367 §3.2] [arxiv:2203.14367 fig. 3]. The paper
does not state the keypoint network's architecture [arxiv:2203.14367 §3.1].
Card: [Thin-Plate Spline Motion Model for Image Animation](/papers/2203.14367).

### Heatmap moments for articulated parts

Each region is a softmax heatmap over pixels. Its centre is the heatmap-weighted
mean of pixel coordinates, and its covariance around that centre is decomposed
by SVD. The linear part of the region transform is U S^(1/2), with the centre as
translation [arxiv:2104.11280 eq. 5] [arxiv:2104.11280 eq. 6]
[arxiv:2104.11280 eq. 7] [arxiv:2104.11280 §3.2]. Shear is not represented, so
each region has five degrees of freedom rather than six [arxiv:2104.11280 §3.2].
A region's motion is composed through a reference frame, and the dense flow is
a per-pixel softmax mix of the K region motions and one background motion
[arxiv:2104.11280 eq. 3] [arxiv:2104.11280 eq. 8] [arxiv:2104.11280 §3.4].
The background is a global 2x3 affine map from its own encoder
[arxiv:2104.11280 §3.3].

The paper argues that regressing local affine parameters needs higher-capacity
networks and generalises poorly [arxiv:2104.11280 §3.2] [arxiv:2104.11280 §4.1].
Its support for this is a toy rotated-rectangles experiment, where the PCA head
beats a regression head across training-set sizes from 32 to 1024. The authors
say the gap exceeds an order of magnitude at the large-data end, and no table
reports those numbers [arxiv:2104.11280 §4.1] [arxiv:2104.11280 fig. 5].

Shape is separated from pose by an animation-via-disentanglement network: two
encoders map the source regions (shape) and driving regions (pose) to 64-dimensional
codes, and a decoder combines them [arxiv:2104.11280 §3.6]. The main model is
trained end to end with a multi-scale VGG-19 reconstruction loss and an
equivariance loss [arxiv:2104.11280 §3.5] [arxiv:2104.11280 eq. 9]
[arxiv:2104.11280 eq. 10]. The shape-pose network is trained afterwards
[arxiv:2104.11280 §3.6]. Card: [Motion Representations for Articulated Animation](/papers/2104.11280).

### Feature-norm points for point-cloud registration

This variant is not heatmap-based. The embeddings come from a DGCNN, which a
Transformer then contextualises across both clouds [arxiv:1910.12240 §3.2].
The keypoints are the k points with the largest embedding norm, taken
separately in each cloud [arxiv:1910.12240 eq. 9]. Keypoints from the two
clouds are matched with a Gumbel-Softmax over feature scores, at a temperature
predicted from the features. The gradient is a straight-through estimate
[arxiv:1910.12240 eq. 10] [arxiv:1910.12240 eq. 11]. The rigid motion is the
SVD closed form on the matched keypoints [arxiv:1910.12240 eq. 2]
[arxiv:1910.12240 eq. 8]. Training runs three passes, moving the source after
each, and the paper uses k = 512 points in training [arxiv:1910.12240 §3.2]
[arxiv:1910.12240 supp. Training Protocol]. The paper says the keypoints are
learned without supervision [arxiv:1910.12240 §4.3]. Its ModelNet40 partial
split reports rotation MSE 10.235 against 45.005 for DCP-v2 [arxiv:1910.12240
tab. 1]. Card: [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240).

## When to use which

The first rows are for image animation. The registration row is last.

| Situation | Use | Why |
|---|---|---|
| Only point positions are needed, or the output head must stay small | Position-only keypoints | No Jacobian head and no inversion. The only measured version is the ablation baseline, which is worse on every metric, but it also drops other terms [arxiv:2003.00196 tab. 1] |
| Points must carry local scale and rotation, and a second equivariance term is acceptable | Keypoints with local Jacobians | Lower AKD and MKR with the Jacobian term than without it, on Tai-Chi-HD [arxiv:2003.00196 tab. 1] |
| Motion is curved or large, and one affine piece per point is the limit, with budget for a TPS fit per group | Keypoint groups with TPS | Starting from MRAA, replacing its affine motion with TPS lowers Tai-Chi-HD AKD from 5.41 to 4.96 [arxiv:2203.14367 tab. 4]. With motion size matched, it has lower AKD at K = 2, 4 and 8 [arxiv:2203.14367 tab. 3] |
| Articulated parts (limbs, bodies), where each part's shape and orientation matter, and a second training stage is acceptable | Heatmap moments for articulated parts | Author-reported Tai-Chi-HD 256 at K = 10: L1 0.047 and AKD 5.58, against 0.056 and 6.53 for FOMM retrained by the authors with 50% more iterations [arxiv:2104.11280 tab. 2] [arxiv:2104.11280 §4.3]. The authors describe VoxCeleb faces as on par with FOMM [arxiv:2104.11280 tab. 2] |
| Camera pan or global motion would otherwise leak into the parts | Keep the background model of the heatmap variant | Without it, Tai-Chi-HD L1 rises from 0.047 to 0.059 and AED from 0.152 to 0.165, while AKD is flat (5.55 against 5.59) [arxiv:2104.11280 tab. 1] |
| Identity fidelity (AED) is the main concern | Check per dataset | The TPS paper's AED is slightly worse on Tai-Chi-HD (0.151 against 0.149) and TED-talks (0.124 against 0.116), and better on VoxCeleb (0.125 against 0.136) [arxiv:2203.14367 tab. 1] |
| Source and driving poses differ a lot | Check before relative transfer | Relative transfer assumes similar poses [arxiv:2003.00196 §3.4] [arxiv:2104.11280 §3.6]. The TPS paper says its method does not work well for an extreme identity mismatch [arxiv:2203.14367 §4.2] [arxiv:2203.14367 §5]. No measured result covers large pose mismatch |
| Test-time transfer must be a closed-form offset | Local Jacobians | Relative transfer is an offset from the first driving frame, applied to each point and Jacobian [arxiv:2003.00196 eq. 14] [arxiv:2003.00196 eq. 15]. The TPS paper uses an animation mode with separate shape and pose encoders [arxiv:2203.14367 §3.4] |
| Training budget is tight | Either, after measuring | FOMM reports about two days on two TitanX GPUs [arxiv:2003.00196 app. B]. The TPS paper reports about three days on two RTX 3090 for Tai-Chi-HD [arxiv:2203.14367 §4.2]. The heatmap paper trains 3M source-driving pairs for the main model and 1M for the shape-pose network [arxiv:2104.11280 app. C]. Hardware and schedules differ, so these are not a like-for-like comparison |
| You need a number to compare against | Author-reported results only | No independent replication is in this wiki. AKD and MKR depend on external pose and landmark detectors in the first two papers [arxiv:2003.00196 §4] [arxiv:2203.14367 §4.2] |
| Input is two partial 3D point clouds with an unknown rigid motion | Feature-norm points with Gumbel matching and SVD | The only point-cloud variant here. Its ModelNet40 numbers are synthetic-only and author-reported [arxiv:1910.12240 §4.1] [arxiv:1910.12240 tab. 1]. Real scans need inference-time fine-tuning, and the paper gives only qualitative results for them [arxiv:1910.12240 §4.2] |

## Implementation sketch

The sketch covers the detector, the equivariance term on positions, and
relative transfer at test time, with the Jacobian head shown as an option.
The TPS block shows where the image TPS variant differs. The region block
shows the heatmap-moment variant. The last block is the point-cloud keypoint
selection. Each step is labelled with its source in the paper.

```pseudocode
# Keypoint detector, position-only or with local Jacobians  [arxiv:2003.00196 §3.1]
# Input:  frame (B, 3, H, W), resized internally to 64 x 64
# Output: positions (B, K, 2), Jacobians (B, K, 2, 2)

def detect(frame, use_jacobian):
    # One heatmap per keypoint, normalised over pixels  [arxiv:2003.00196 §3.1]
    heatmaps = softmax(keypoint_net(frame))   # (B, K, 64, 64)
    # Position is the expected grid coordinate under its heatmap  [§3.1]
    positions = sum(heatmaps * grid)          # (B, K, 2)
    if not use_jacobian:
        # Position-only variant: Jacobian fixed to identity  [§3.1]
        return positions, identity(B, K)
    # Four extra channels per keypoint, same weights as the heatmap  [§3.1]
    jac_maps = jacobian_net(frame)            # (B, K, 4, 64, 64)
    jacobians = sum(heatmaps * jac_maps)      # (B, K, 2, 2)
    return positions, jacobians
```

```pseudocode
# Equivariance term on keypoint positions  [arxiv:2003.00196 §3.3]
# Input:  frame D, a sampled warp T (affine + thin-plate spline)
# Output: scalar loss

def position_equivariance(D, T):
    p, _ = detect(D, use_jacobian=False)
    p_warped, _ = detect(T(D), use_jacobian=False)
    # Warped detections, mapped back, should match the originals  [eq. 10] [eq. 11]
    return l1(p, T_points(p_warped))
```

```pseudocode
# Relative transfer at test time  [arxiv:2003.00196 §3.4]
# Input:  source keypoints, driving keypoints, driving keypoints at frame 1

def transfer(kp_source, kp_driving, kp_driving_first):
    # Move each source keypoint by the driving motion since its first frame
    return kp_source + (kp_driving - kp_driving_first)
```

```pseudocode
# TPS variant: one thin-plate spline per keypoint group  [arxiv:2203.14367 §3.1]
# Input:  K groups of 5 keypoints, driving and source, each (K, 5, 2)
# Output: K warp grids, plus one affine background grid

def fit_tps_fields(kp_driving, kp_source):
    fields = []
    for k in range(K):
        # Smoothest map through the five pairs  [eq. 1] [eq. 2] [eq. 3]
        # Direction of the fit: see Pitfalls
        fields.append(solve_tps(kp_driving[k], kp_source[k]))
    return fields
```

```pseudocode
# Per-pixel weights over K TPS warps plus the background  [eq. 5] [eq. 6]
# Output: flow, the weighted sum of the K+1 warps

def combine(warps, logits):
    weights = softmax(logits, dim=parts)      # (K+1, H, W)
    return sum(weights[j] * warps[j] for j in range(K + 1))
```

```pseudocode
# Region variant: centre and linear part from heatmap moments  [arxiv:2104.11280 §3.2]
# Input:  image (B, 3, h, w) at quarter resolution  [arxiv:2104.11280 app. C]
# Output: centres (B, K, 2), linear parts (B, K, 2, 2)

def measure_regions(image):
    # Softmax over pixels, one heatmap per region; no temperature in the paper
    heatmaps = softmax(region_net(image), dim=pixels)  # (B, K, h, w)
    # Centre is the heatmap-weighted mean of coordinates  [eq. 5]
    centres = sum(heatmaps * coords)                    # (B, K, 2)
    # Covariance of each heatmap around its centre  [eq. 6]
    offsets = coords - centres
    covars = sum(heatmaps * outer(offsets, offsets))    # (B, K, 2, 2)
    # Principal axes by SVD; linear part is U S^(1/2)  [eq. 6] [eq. 7]
    # The sign of each axis is not fixed by the paper; see Pitfalls
    U, S, _ = svd(covars)
    linear = U @ diag(sqrt(S))                          # (B, K, 2, 2)
    return centres, linear
```

```pseudocode
# Point-cloud keypoints by feature norm, no keypoint labels  [arxiv:1910.12240 eq. 9]
# Input:  points (B, 3, N), per-point embeddings (B, d, N), k
# Output: keypoints (B, 3, k), keypoint embeddings (B, d, k)

def top_k_by_norm(points, feat, k):
    # Score each point by the L2 norm of its embedding  [eq. 9]
    score = norm(feat, dim=channels)          # (B, N)
    # Keep the k highest-scoring points  [eq. 9]
    idx = topk(score, k)                      # (B, k)
    return gather(points, idx), gather(feat, idx)
```

## Contradictions

The three image-animation papers report Tai-Chi-HD numbers that cannot be
compared directly. Each paper reports results for its own model, with its own
baselines and training schedule. For the same dataset, the first paper reports
its full model at L1 0.063, AKD 6.862 and MKR 0.036 [arxiv:2003.00196 tab. 1].
The second reports MRAA at L1 0.048, AKD 5.41 and MKR 0.025, and its own
model at L1 0.045, AKD 4.57 and MKR 0.018 [arxiv:2203.14367 tab. 1]. The
second paper says its metrics follow MRAA's evaluation setup
[arxiv:2203.14367 §4.2]. The third reports its own model at L1 0.047, AKD 5.58
and MKR 0.027, and FOMM, retrained by its authors on a longer schedule, at L1
0.056, AKD 6.53 and MKR 0.033 [arxiv:2104.11280 tab. 2] [arxiv:2104.11280 §4.3].
The third paper's FOMM row is lower than the first paper's own full model, which
is the same method. The cards do not establish why; the schedule and evaluation
setup are the likely differences, but neither is tested here. No wiki card runs
the three methods head to head, so no result settles the others.

The third paper also conflicts in part with the first on regressed Jacobians.
The first reports that its regressed local affine terms, with their own
equivariance term, lower AKD and MKR on Tai-Chi-HD [arxiv:2003.00196 tab. 1].
The third argues that regressed affine parameters generalise poorly
[arxiv:2104.11280 §3.2]. Its only isolated evidence is the toy experiment
[arxiv:2104.11280 §4.1]. Its FOMM comparison changes the region representation
and the background model together, so it does not test the head choice alone.
Without its background model, the third paper's method has higher L1 than
retrained FOMM (0.059 against 0.056) but lower AKD (5.55 against 6.53)
[arxiv:2104.11280 tab. 1].

PRNet does not conflict with the image papers. Its metric is rotation error
on point clouds, and no metric is shared with them [arxiv:1910.12240 tab. 1]
[arxiv:2003.00196 tab. 3]. Its numbers are not comparable to theirs.

## Pitfalls

- **K means different things.** In the first image paper K = 10 is ten points,
  each with a Jacobian [arxiv:2003.00196 §4]. In the second, K = 10 is ten
  groups of five points, each group defining one TPS [arxiv:2203.14367 §3.1].
  In the third, K = 10 is ten regions, each a heatmap with its own centre,
  extent and orientation [arxiv:2104.11280 §3.2] [arxiv:2104.11280 tab. 2].
  Compare motion sizes by dimension, not by K. The first paper uses 60 floats
  for K = 10 [arxiv:2003.00196 §4].
- **Motion dimension in the second paper.** Its formula gives 134 for K = 8,
  while its text gives 118 [arxiv:2203.14367 §4.3]. Check which holds before
  you quote the size of its comparison.
- **Heatmap form (first paper).** As printed, eq. 6 grows with distance, so it
  is not a bump-shaped heatmap. The released code uses a Gaussian
  [arxiv:2003.00196 eq. 6] [code: modules/dense_motion.py#L37-L39].
- **Softmax temperature (first paper).** The paper does not state it. The
  shipped configs use 0.1 [code: config/taichi-256.yaml#L43]. The third paper
  also states no temperature for its heatmap softmax [arxiv:2104.11280 §3.2].
- **Jacobian inversion (first paper).** Relative and Jacobian transfer invert
  the driving Jacobian. The paper assumes local bijectivity, and the code has
  no explicit guard against near-singular matrices [arxiv:2003.00196 §3.1]
  [code: modules/dense_motion.py#L56].
- **Relative or absolute transfer (first paper).** The Fig. 1 caption says
  absolute transfer is used for Tai-Chi-HD, but the shipped config uses
  relative transfer [arxiv:2003.00196 §3.4] [code: config/taichi-256.yaml#L144-L147].
- **Direction of the TPS constraint (second paper).** The paper writes the
  constraint with the source keypoints mapped to the driving ones
  [arxiv:2203.14367 eq. 1]. The released code fits the other way and uses the
  result as the sampling grid [code: modules/util.py#L20-L22]. Pick one
  direction and check it against the sampling step.
- **TPS numerics (second paper).** The released code adds a 0.01 ridge to the
  linear system and an epsilon inside the logarithm of the radial term. The
  paper gives exact interpolation and no epsilon [arxiv:2203.14367 eq. 3]
  [code: modules/util.py#L41-L42].
- **Dropout rescale (second paper).** Eq. 7 divides the kept terms by 1 − P.
  The released code does not apply that division to the exponentials, so a
  literal reading and the code train differently. Pick one and say which
  [arxiv:2203.14367 eq. 7] [code: modules/dense_motion.py#L109-L116].
- **Equivariance input (second paper).** Eq. 10 uses the source image, while
  the code applies the random warp to the driving frame
  [arxiv:2203.14367 eq. 10] [code: modules/model.py#L149-L160].
- **Loss scale (second paper).** The text gives an unweighted sum of absolute
  values [arxiv:2203.14367 eq. 14]. The released code takes means and weights
  each term by 10 [code: config/taichi-256.yaml#L100-L108].
- **Hidden hyperparameters (second paper).** The paper does not give the
  learning rate, optimiser, batch size, dropout probability, pyramid scales
  or the architecture of its three modules [arxiv:2203.14367 §3]. Those values
  must come from the code or be chosen by the implementer.
- **Axis sign (third paper).** The SVD returns each principal axis only up to
  sign, and the paper gives no sign rule for it [arxiv:2104.11280 §3.2]. An
  implementation has to choose one. The paper does not test what happens
  without one.
- **Shape-pose scaling (third paper).** Training randomly scales the driving
  regions per axis, which pushes shape to come from the source branch. The
  paper gives no range for the scale [arxiv:2104.11280 §3.6].
- **Table inconsistency (third paper).** The Tai-Chi-HD 256 full-model L1 at
  K = 10 is 0.047 in tab. 1 and tab. 2, but 0.048 in tab. 4. Its AKD is 5.59
  in tab. 1 and tab. 4 and 5.58 in tab. 2 [arxiv:2104.11280 tab. 1]
  [arxiv:2104.11280 tab. 2] [arxiv:2104.11280 tab. 4]. Use the value from the
  table you cite.
- **Region count is a tuning choice (third paper).** From K = 5 to K = 20,
  the paper's own AKD on Tai-Chi-HD falls from 6.04 to 5.17, while retrained
  FOMM's AKD rises from 7.34 to 8.29 [arxiv:2104.11280 tab. 1]. Choose K on
  validation data, not from these rows alone.
- **Keypoint count is fixed in the image papers.** All three image papers use
  K = 10 for their main results [arxiv:2003.00196 §4] [arxiv:2203.14367 §4.2]
  [arxiv:2104.11280 tab. 2]. The first paper links a baseline's failure on
  large pose changes to keypoints missing from the arms [arxiv:2003.00196 §4].
- **Keypoint count matters for registration.** PRNet's ablation lowers
  accuracy sharply as k falls: MAE(R) is 27.843 at k = 16 and 1.454 at
  k = 512 [arxiv:1910.12240 supp. (d)]. The test-time k is not stated
  [arxiv:1910.12240 §4.1].
- **PRNet keypoints are not part-aligned.** They are picked by feature norm,
  with no keypoint loss. The paper shows them in figures only, so do not read
  them as object parts [arxiv:1910.12240 §4.3] [arxiv:1910.12240 fig. 4]
  [arxiv:1910.12240 fig. 5].
- **PRNet matching is not one-to-one.** The conclusion says the matchings are
  not bijective. A fix (Gumbel-Sinkhorn) is future work [arxiv:1910.12240 §5]
  [arxiv:1910.12240 fig. 6].
- **PRNet temperature has no stated range.** The paper gives none for λ, so
  the predicted temperature is unbounded as written [arxiv:1910.12240 §3.2].
  The released code clamps it [code: model.py#L411].
- **PRNet test-time matching is random.** The released code samples the
  Gumbel matching at inference as well [code: model.py#L441]. The paper does
  not say whether it averages over samples or fixes a seed.
- **PRNet gradient is biased.** The straight-through estimator is biased, and
  the paper gives no variance or convergence comparison [arxiv:1910.12240 §3.2].
- **Evaluation depends on third-party detectors.** AKD and MKR use an external
  landmark or pose detector in both image papers, so changing it changes the
  numbers [arxiv:2003.00196 §4] [arxiv:2203.14367 §4.2].
- **Human judgements are thin in the second paper.** It uses 20 video pairs and
  does not report the number of raters or a significance test
  [arxiv:2203.14367 §4.2] [arxiv:2203.14367 tab. 2]. The first paper uses 50
  input pairs with 10 raters per question [arxiv:2003.00196 §4].
- **TED-talks data licence (third paper).** The source videos were shared under
  CC BY-NC-ND 4.0, which is non-commercial and no-derivatives, and the authors
  cut and rescaled them [arxiv:2104.11280 app. E]. The dataset is not in this
  wiki, and its numbers are author-reported. Check those terms before you
  redistribute or train on that data.
- **Costs.** The first image paper reports about 100k iterations at batch 20
  to 40, roughly two days on two GPUs for Tai-Chi-HD and VoxCeleb
  [arxiv:2003.00196 app. B]. Each TPS fit solves an (N+3) × (N+3) linear system
  per group and per sample [arxiv:2203.14367 eq. 2] [arxiv:2203.14367 eq. 3].
  The second paper reports about eight days of training on TED-talks
  [arxiv:2203.14367 §4.2]. PRNet reports about 10 hours on four V100 GPUs
  [arxiv:1910.12240 supp. Training Protocol].

## Papers

- [First Order Motion Model for Image Animation](/papers/2003.00196): learns ten keypoints with local 2x2 Jacobians from video reconstruction and an equivariance loss, without object annotations.
- [Thin-Plate Spline Motion Model for Image Animation](/papers/2203.14367): learns ten groups of five keypoints and warps with one thin-plate spline per group, plus multi-resolution occlusion masks.
- [Motion Representations for Articulated Animation](/papers/2104.11280): measures ten heatmap regions by their mean and covariance (SVD), adds a global affine background and a shape-pose network, without part labels.
- [PRNet: Self-Supervised Learning for Partial-to-Partial Registration](/papers/1910.12240): selects the top-k points by feature norm in two partial point clouds, matches them with Gumbel-Softmax and solves the rigid motion with SVD, without keypoint labels.