Gait Recognition in the Wild with Dense 3D Representations and A Benchmark

In plain words

Gait recognition tries to tell people apart by the way they walk, from video. Most systems look at a flat 2D outline of the walker, which throws away how the body is shaped and which way the person is turned relative to the camera. Cameras in a real building see people from many angles, so those systems do poorly there.

This paper adds a 3D model of the human body, estimated from each video frame, as a second input. A small network reads the 3D pose, body shape and camera settings for each frame and produces a transform. The appearance features from the outline are multiplied by that transform, so that the same person seen from different angles lines up better. The authors also built a benchmark, Gait3D, with 4,000 people filmed by 39 cameras in a supermarket, and they report that the method beats earlier 2D approaches on it. All numbers are the authors' own.

TL;DR

• Two-branch network: a silhouette CNN for appearance and a 3D spatial transformation network driven by SMPL body parameters.
• The transform is a per-frame matrix applied to the appearance feature map, meant to normalise viewpoint and shape.
• Trained end to end with a triplet loss plus a cross-entropy loss, and evaluated with cosine similarity.
• Headline (author-reported): 53.20% Rank-1 on Gait3D at 88x128 input; 46.30% at 44x64. The 3D branch adds about 5.5 points over the same network without it.
• The dataset is not public; access needs a signed agreement, and the repo asks for non-commercial use.

Problem

Gait recognition is usually built on 2D representations such as silhouettes, gait energy images or 2D skeletons §1. The paper argues that projecting the 3D body onto a plane discards viewpoint, shape and dynamics, and that 2D methods fail in unconstrained scenes with varied camera angles and occlusion §1 §2. Existing benchmarks are mostly lab or small campus scenes §2 tab. 1. No dataset then offered dense 3D representations of walkers in the wild §2.

The method assumes that per-frame 3D human meshes (SMPL parameters) can be recovered from ordinary video §1. The paper uses ROMP for this on its own data §4.2.3.

Method

The model has two branches, each run over the frames of one sequence §3.1.

Appearance branch (SLN). A CNN with six convolution layers, modelled on the GaitSet backbone, maps each binary silhouette to a feature map §3.2 eq. 1. The layers and their channel counts are in app. A.1, tab. 3. Every convolution is followed by LeakyReLU with slope 0.01, and max pooling follows the second and fourth convolutions app. A.1, tab. 3.

Viewpoint and shape branch (3D-STN). The SMPL vector of each frame has D = 85 values: 24 x 3 pose values, 10 shape values and 3 camera values §3.2. Three fully connected layers with widths 128, 256 and h x w map this vector to a transform vector g_i §3.2 eq. 2. Each layer uses batch normalisation and ReLU; dropout is applied to the last two layers §3.2. Dropout rates are 0.0, 0.2 and 0.2 app. A.1, tab. 4.

Alignment. The transform vector is reshaped into a square matrix G_i. The feature map is zero-padded on its short edge to a square, and then multiplied by (I + G_i), where I is the identity §3.2 eq. 3. The paper's appendix sets h = w = max(h, w) app. A.1.

Aggregation. The aligned maps are pooled over the frames with Set Pooling, and then Horizontal Pyramid Pooling from GaitSet produces the sequence feature §3.2. The code splits the features into 31 horizontal parts (bins 16, 8, 4, 2, 1) and gives each part its own embedding and BNNeck head smplgait_gait3d.yaml smplgait.py.

Training and inference. The two branches are trained end to end on L = alpha L_tri + beta L_ce, with a triplet loss and a cross-entropy loss §3.3 eq. 4. Alpha is 1.0 and beta is 0.1 §5.1.3. At inference, the query and gallery sequences are compared with cosine similarity §3.3.

The pseudocode below follows the paper text and the released model code. It is not an algorithm box from the paper.

# SMPLGait training step, reconstructed from §3.2-§3.3 and the repo code
# Input:  silhouettes (B, S, H, W); smpls (B, S, 85); labels (B,)
# Output: scalar loss
# B identities, 4 sequences each; S = 30 frames [§5.1.3]

def training_step(silhouettes, smpls, labels):
    # 1. Per-frame appearance maps F_i from six conv layers [§3.2] [eq. 1]
    maps = silhouette_cnn(silhouettes)              # (B, S, C, h, w)

    # 2. Per-frame transform vector g_i from the SMPL vector
    #    three FC layers, widths 128, 256, h*w [§3.2] [eq. 2] [app. A.1]
    vectors = smpl_transform_net(smpls)             # (B, S, n*n)

    # 3. Reshape g_i into a square matrix G_i and add the identity [§3.2]
    transforms = reshape_square(vectors) + identity # (B, S, n, n)

    # 4. Zero-pad each map to a square, then compute F_i (I + G_i) [eq. 3]
    padded = zero_pad_to_square(maps)               # (B, S, C, n, n)
    aligned = padded @ transforms[:, :, None]       # (B, S, C, n, n)

    # 5. Pool over frames, then horizontal pyramid pooling [§3.2]
    #    bins 16, 8, 4, 2, 1 = 31 parts [code: smplgait_gait3d.yaml#L63-L68]
    pooled = max_over_frames(aligned)               # (B, C, n, n)
    parts = horizontal_pyramid(pooled, [16, 8, 4, 2, 1])  # (B, C, 31)

    # 6. One FC per part, then BNNeck logits [code: smplgait.py#L84-L86]
    embeddings = per_part_fc(parts)                 # (B, 256, 31)
    logits = bn_neck(embeddings)                    # (B, classes, 31)

    # 7. Triplet term on embeddings, cross-entropy on logits [§3.3]
    loss_tri = triplet(embeddings, labels)          # margin 0.2 [code]
    loss_ce = cross_entropy(logits, labels)         # scale 16 [code]

    # 8. Weighted sum, alpha = 1.0 and beta = 0.1 [eq. 4] [§5.1.3]
    return 1.0 * loss_tri + 0.1 * loss_ce

Equation numbers follow the LaTeX order: the four displayed equations are eq. 1 to eq. 4. Table and figure numbers were counted from the LaTeX source, because the arXiv PDF was not read; if a number looks off, check the section instead.

Hyperparameters

Results

Author-reported. Setup: Gait3D, trained on 3,000 identities and tested on 1,000 identities with the split in §4.3. Metrics are Rank-1, Rank-5, mAP and mINP in percent. Hardware is not stated. Baseline numbers come from the authors' runs; the paper says GaitSet, GaitPart, GLN and GaitGL use OpenGait implementations §5.1.1.

Full comparison from tab. 2:

Source: tab. 2. PoseGait and GaitGraph are skeleton-based; the paper reports one column for them tab. 2 caption.

Cross-domain, GaitSet as the backbone, from the appendix tab. 5. GREW's test set is not public, so the authors evaluated on 1,000 sampled GREW training identities §B.2.1:

Other reported findings, all author-reported:

• Ablation: removing the 3D branch (SMPLGait w/o 3D, which the footnote equates with the OpenGait baseline) drops Rank-1 by 5.5 points at 88x128 and 3.4 points at 44x64 tab. 2 §5.1.5.
• Input size: most methods improve at 88x128 over 44x64; GaitGL is the exception, which the authors attribute to its 3D-CNN backbone §5.2 tab. 2.
• Training frames: performance rises and then falls as training sequences grow from 10 to 50 frames, with the best around 30 §5.2 fig. 5.
• Training identities: performance rises steadily from 0.5K to 3K training identities §5.2 fig. 6.
• Test frames: accuracy rises as more of each test sequence is used, from 10% to 100% §B.1 fig. 7.
• Visual results: good cases, occluded cases, low-quality silhouettes, and one failure with similar clothing and shapes §B.3 fig. 8 to fig. 11.

Paper vs code

Repository: Gait3D/Gait3D-Benchmark, commit cbbafe7c28b11da85aaabb2b5a4303503c82f70c. The paper's abstract says the code is at gait3d.github.io. The project page links this repository [WebFetch of gait3d.github.io]. The repository's README names Jinkai Zheng and Xinchen Liu, two of the paper's authors, as maintainers README.md. The repository has no LICENSE file, so the code is cited by path and line only, and no code is quoted.

Discrepancies found:

1. Two configs disagree. Paper: cosine similarity and Adam §3.3 §5.1.1. train.sh trains configs/smplgait/smplgait_gait3d.yaml: cosine, Adam, 180,000 iterations train.sh smplgait_gait3d.yaml [L73] [L93]. But datasets/Gait3D/README.md points to configs/smplgait/smplgait.yaml, which uses Euclidean distance, SGD with lr 0.1 and 60,000 iterations README.md smplgait.yaml [L71-L73] [L94]. The paper does not say which one produced its numbers.

2. Schedule. Paper: 1,200 epochs, lr x0.1 at epochs 200 and 600 §5.1.1. Code: 180,000 iterations, lr x0.1 at iterations 30,000 and 90,000 smplgait_gait3d.yaml [L93]. The epoch count cannot be converted to iterations, because the number of iterations per epoch is not stated.

3. Input resolution. Headline numbers are at 88x128 tab. 2. The shipped config uses BaseSilCuttingTransform with img_w: 64, which crops 10 pixels from each side of the 64-px silhouettes, giving 44 wide by 64 high smplgait_gait3d.yaml transform.py. All three SMPLGait configs use img_w: 64. The merge command in the docs uses only the 64-px silhouettes pretreatment.md README.md. The model zoo lists a 128-px checkpoint and its 53.20 result model_zoo.md, but no 128-px config or merge step is in the repo.

4. Feature grid is fixed at 16 x 16. The paper sets n = max(h, w) app. A.1. The code hard-codes 16 x 16 for the transform matrix and its reshape, and for the final layer width of 256 smplgait.py [L50-L55] [L73-L77]. My arithmetic says a 64-px input gives a feature height of 16 after two 2x2 poolings, so the hard-coded size matches the 44 x 64 setting only. This is an inference from shapes, not something the repo states.

5. Padding. The paper pads both F and G to a square §3.2. The code pads only the feature map, with zero columns up to the height smplgait.py. G is 16 x 16 by construction.

6. Dropout placement. The paper lists dropout on the last two FC layers, with batch norm and ReLU after each layer §3.2. The code applies dropout before batch norm in layers 2 and 3 smplgait.py.

7. SMPL vector order. The paper lists pose, shape, camera §3.2. The preprocessing concatenates camera (3), pose (72), shape (10) pretreatment_smpl.py. The length, 85, is the same; the order matters if you assemble vectors by hand.

8. Loss details not in the paper. The code uses a triplet margin of 0.2, a cross-entropy logit scale of 16 and label smoothing of 0.1 smplgait_gait3d.yaml ce.py. The triplet loss is a batch-all version that averages the non-zero triplets triplet.py, and it is applied to the embeddings before BNNeck smplgait.py.

9. Temporal pooling. The paper calls it Set Pooling, following GaitSet, and does not give the operator. The code takes a max over frames smplgait.py.

10. Convolutions have no bias. BasicConv2d is built with bias=False, and the backbone has no normalisation. The paper does not say either modules.py plain.py.

11. Query selection. Paper: one random sequence per identity as the query §4.3. The code evaluates against a fixed PROBE_SET of 1,000 sequences in the released split file Gait3D.json evaluator.py. The paper gives no seed, so the random choice cannot be reproduced; the fixed list is what the code uses.

12. Baselines. The paper says the GaitSet, GaitPart, GLN and GaitGL numbers come from OpenGait §5.1.1. The repo has configs for them model_zoo.md. I did not check those configs against the paper.

Pitfalls

• The dataset is not public. The project page says it is available only after signing an agreement [WebFetch of gait3d.github.io]. The repo README asks for non-commercial use README.md. The dataset README says the original data is not publicly available README.md. Do not redistribute it or use it commercially.
• Raw frames are not released. Only silhouettes, SMPL parameters, 3D poses and 2D poses are released §4.2.3. To reproduce the pipeline on other video you must run ROMP, HRNet segmentation and HRNet pose yourself. The paper gives no costs for this.
• Dataset construction needs manual work. Human annotators removed sequences with more than one person, filtered outliers within groups, and labelled 4,000 boxes and 4,000 images for fine-tuning §4.2.1 §4.2.2. The DBSCAN parameters are not stated §4.2.2.
• Supplementary material is missing from the source. The paper defers details to the supplementary material for the framework, experiments and limitations §3.2 §5.2 §6 app. C. Those details are unknown here.
• Short windows. Training uses 30 frames per sequence. Accuracy fell when training sequences were longer than about 30 frames, which the authors attribute to noisy speed and route changes; they also note that temporal dynamics are not fully explored fig. 5 §5.2 app. C.
• Viewpoint alignment is in feature space. The transform is a learned matrix applied to the feature map; the meshes are not rotated §1 §3.2. The paper's claim that this normalises viewpoint is the authors' interpretation.
• Privacy and misuse. The authors discuss privacy, bias, and possible misuse for surveillance, and limit the dataset to research use under a strict licence §6 app. C.
• Cross-domain protocol is not reproducible. GREW's test set is not public and the 1,000 identities were sampled randomly §B.2.1. The repo's model zoo lists 15.80 for the official GREW split and 16.50 for the authors' own split model_zoo.md. Use the split, not the paper's sampled numbers, if you compare.

Lineage

• Builds on: GaitSet (AAAI 2019; arXiv 1811.06186 per the repo's model zoo link) for the silhouette backbone, set pooling and horizontal pyramid pooling §3.2 §5.1.1. Also SMPL [tog/LoperM0PB15] and ROMP for body-mesh recovery §2 §4.2.3.
• Compared against: GEINet, GaitSet, GaitPart, GLN, GaitGL, CSTL, PoseGait and GaitGraph tab. 2.
• Later work by the same group, listed in the repo README and not compiled here: MTSGait (arXiv 2209.00355), ParsingGait (arXiv 2308.16739) and XGait (arXiv 2411.10742) README.md.