Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization

In plain words

Person re-identification asks whether two photos from different cameras show the same person. It gets hard when someone changes clothes between sightings, because the most obvious cue, the colour and cut of the outfit, no longer matches. Other body cues such as face or shape are hard to read from a single image.

This paper uses the way a person walks, their gait, as a hidden teacher during training. A second network, the gait stream, is given one silhouette of the person and learns to imagine a short sequence of walking frames, then to summarise the walking pattern. The main image network is nudged to produce features that agree with the gait stream's view of the same person. The gait stream is then thrown away, so the deployed model is still a plain image model that takes one picture and returns one feature vector.

TL;DR

• Image-only re-identification fails when clothes change; this paper adds gait as a training-time regulator for an image model. §1 §2
• A gait sequence predictor expands a single silhouette into several frames, which a gait recogniser turns into features. §3.1
• Two feature-level losses tie the gait and image streams together; the gait stream is discarded at inference. §3.3 tab. 7
• Author-reported gains on LTCC, PRCC, Real28 and VC-Clothes; the released code contains only the image baseline. §4.5 tab. 8

TL;DR

Read this card when the target is image-based person re-identification under clothing change. The method only changes training: a gait-prediction stream and two consistency losses are added, and the test-time model is the ReID backbone alone. Results are author-reported and depend on the backbone. The official repository does not contain the gait code, so the method must be implemented from the paper, with the gaps in the Pitfalls section kept in mind.

Problem

Standard re-identification models assume that query and gallery images of one person show the same clothes §1. On datasets where clothes change over days, accuracy drops sharply §1. Existing clothing-change methods rely on contour sketches, body shape or face and hairstyle cues, which are prone to estimation error from a single image §1 §2. Gait-recognition methods normally need video sequences and do not transfer directly to single images, where viewpoint change and occlusion are harder §1 §2 tab. 1. The paper targets the single-image case, because video adds latency and storage cost for real-time surveillance §1.

Method

The model has two streams trained jointly and one stream used at test time §3 fig. 2.

Auxiliary Gait-Stream (training only) §3.1. A person silhouette is first extracted by a segmentation model; the paper names PointRend as an example §3. The gait stream has two parts.

1. Gait Sequence Prediction (GSP). An encoder E maps the silhouette to a compact feature eq. 1. A position embedder P predicts the position of the input frame within the sequence, and this is trained with a squared-error position loss eq. 2. A fully connected aggregator A concatenates the feature with the predicted position and maps the result back to the feature size eq. 3. A decoder D then predicts N frames, and the prediction loss compares them with the ground-truth sequence eq. 4. The paper's choice is N = 8 tab. 6.
2. Gait feature extraction. The predicted frames are passed to GaitSet, a set-based gait recogniser, which produces a gait feature g §3.1.

The "middle frame input principle" is the key design choice in GSP. During GSP training the input is always the middle frame of the ground-truth sequence, so the predictor learns to produce frames both before and after the input §3.1. Input position is therefore a training label, not something known at test time §3.1 tab. 6b.

Main ReID-Stream §3.2. The image backbone can be any standard network; the paper lists ResNet-50, PCB, MGN and OSNet as examples §3.2 and evaluates ResNet-50, OSNet, LTCC-shape and PRCC-contour as backbones §1 §4.5. The ReID feature r is trained with a classification loss and a triplet loss with batch-hard mining §3.2.

Joint learning with the Semantics Consistency (SC) constraint §3.3. Features r (image) and g (gait) pass through embedding layers that give r̂ and ĝ of the same dimension §3.3. Two losses link them.

• A moment-matching loss compares the mean and the σ term of the two embeddings eq. 5. The paper calls this an MMD loss; the formula matches only first and second moments, not a kernel MMD (see Pitfalls).
• A reconstruction loss passes r̂ and ĝ through reconstruction layers and penalises the error against the original r and g eq. 6.

The gait stream is not used at inference. The ReID vector r is the output. Of the options tested, only the aligned features r̂ + ĝ score higher, and they need the gait stream at test time tab. 7.

Training pipeline. Three phases §3.3 alg. 1 app. 2:

1. pre-train GaitSet on the gait dataset CASIA-B;
2. jointly train GSP and GaitSet on CASIA-B, first GSP alone then both;
3. jointly train the gait stream and the ReID stream on the cloth-changing dataset with the SC losses.

Phase 3 cannot use ground-truth gait sequences, because the re-identification datasets do not have them. The gait stream's prediction loss is replaced by an L1 distance between the input mask and the middle predicted frame app. 2.

# Phase 1: pre-train GaitSet on a gait dataset  [alg. 1] [app. 2]
# Batch: P = 16 identities x N = 8 frames of 64 x 44 silhouettes
# Adam, learning rate 1e-4, 80 epochs, separate triplet margin 0.2
def gaitset_pretrain_step(silhouette_sets, labels):
    # Separate triplet loss over the gait features  [§3.1] [app. 2]
    gait = gaitset(silhouette_sets)                 # (P, N, d_g)
    return separate_triplet_loss(gait, labels, margin=0.2)

# Phase 2a: warm up GSP alone, 80 epochs  [alg. 1] [app. 2]
# Batch: P = 4 identities x N = 8 consecutive frames (B = 32)
# Adam, learning rate 5e-4, weight decay 1e-4, lr x 0.1 at epoch 40
def gsp_warmup_step(sequences):
    # Feed the 4th of 8 frames (the middle frame)  [§3.1] [app. 2]
    middle = sequences[:, 3]                        # (B, 64, 64)
    predicted, position = gsp(middle)               # (B, 8, 64, 64)
    # Position loss [eq. 2] and prediction loss [eq. 4], weight 1 : 1
    return position_loss + prediction_loss

# Phase 2b: GSP and GaitSet jointly, 160 epochs  [alg. 1] [app. 2]
# Same batch as Phase 2a; lr 5e-4, weight decay 1e-4, lr x 0.5 at 40, 80, 120
def gsp_joint_step(sequences, labels):
    middle = sequences[:, 3]                        # (B, 64, 64)
    predicted, position = gsp(middle)               # (B, 8, 64, 64)
    gait = gaitset(predicted)                       # (B, d_g)
    # Add the separate triplet loss on gait features, weight 1.0  [app. 2]
    return (position_loss + prediction_loss
            + separate_triplet_loss(gait, labels))

# Phase 3: joint gait and ReID training on a cloth-changing dataset  [alg. 1]
# Batch: P = 10 identities x N = 8 images (B = 80), 240 epochs [app. 2]
# Adam: ReID stream lr 5e-4; gait stream lr 1e-5; weight decay 1e-5
def train_step(images, masks, labels):
    # Resize each 2:1 person mask and zero-pad it to a 64 x 64 square [app. 1]
    silhouettes = resize_and_zero_pad(masks)        # (B, 1, 64, 64)
    # GSP predicts the sequence from the middle silhouette  [§3.1] [eq. 2-3]
    middle = silhouettes                            # (B, 1, 64, 64)
    predicted, position = gsp(middle)               # (B, 8, 64, 64)
    # The input is the middle frame by construction, so its position
    # label is known without a gait sequence  [§3.1] [app. 2]
    middle_position = 8 // 2                        # p_mid = N // 2, N = 8
    # Phase 3 prediction loss: L1 to the middle predicted frame  [app. 2]
    gait_pred_loss = l1(predicted[:, 3], silhouettes)
    gait = gaitset(predicted)                       # (B, d_g)
    reid = backbone(images)                         # (B, d_r)
    logits = classifier(reid)
    # Two SC embeddings and reconstruction layers  [§3.3]
    r_hat = embed_r(reid)
    g_hat = embed_g(gait)
    # Moment matching between embeddings  [eq. 5]
    loss_mmd = (sq_norm(mean(g_hat) - mean(r_hat))
                + sq_norm(sigma(g_hat) - sigma(r_hat)))
    # Reconstruct the original features from the embeddings  [eq. 6]
    loss_recon = (sq_norm(recon_g(g_hat) - gait)
                  + sq_norm(recon_r(r_hat) - reid))
    # Weights: ReID cls 1.0, ReID hard triplet 1.0, gait 0.1, SC 0.5
    # [alg. 1] [app. 2]
    return (1.0 * classification_loss(logits, labels)
            + 1.0 * hard_triplet_loss(reid, labels)
            + 0.1 * (position_loss(position, middle_position)
                     + gait_pred_loss
                     + separate_triplet_loss(gait, labels))
            + 0.5 * loss_mmd
            + 0.5 * loss_recon)

# Inference: only the ReID stream; GSP, GaitSet and SC layers are unused
# [§3.3] [tab. 7]
def embed(image):
    # Use the ReID feature r, not the reconstructed vector  [tab. 7]
    reid = backbone(image)                          # (1, d_r)
    return reid

Hyperparameters

Datasets §4.1 app. 3 tab. 2

Split details from app. 3:

• Real28: 336 query and 3,988 gallery images; used for testing only. Models are trained on VC-Clothes §4.2.
• VC-Clothes: 256 identities for training and 256 for testing, 9,449 training images, 1,020 queries (4 per person per camera) and 8,591 gallery images.
• LTCC: 77 training identities (46 with clothing change, 31 without) and 75 test identities (45 with change, 30 without).
• PRCC: 150 training and 71 test identities. Gallery is one image per test identity from Camera A. Probes are Camera B (same clothes) or Camera C (different clothes).

Results

Author-reported. Setup: ResNet-50 unless stated, or OSNet, or the LTCC-shape and PRCC-contour baselines. Each table is reproduced from the main text §4; the supplement app. 4 and app. 5 add rows not reproduced here.

Real cloth-changing image ReID, GS-GSP ablation tab. 3. Columns are mAP and Rank-1 for each dataset.

Silhouette versus gait on LTCC cloth-changing setting tab. 4. Columns are mAP and Rank-1.

Pure gait recognition on LTCC cloth-changing tab. 5. Columns are mAP and Rank-1.

Design choices on LTCC tab. 6. Columns are mAP and Rank-1, standard then cloth-changing. The paper's setting is N = 8, middle input, MMD and reconstruction on.

(a) Length N of the predicted sequence:

(b) Input frame position in GSP:

(c) SC loss variants:

Inference strategy on LTCC tab. 7. Columns are mAP and Rank-1, standard then cloth-changing.

Main comparison on LTCC tab. 8a. Columns are Rank-1 then mAP for each setting. Standard† and cloth-changing† train only on identities that change clothes. Rows marked "--" are not reported.

PRCC tab. 8b. Cross-clothes setting (Camera A gallery, Camera C probes). Columns are Rank-1, Rank-10, Rank-20. The source typesets one value as "91,67"; it is read here as 91.67.

The same-clothes setting (Camera A and B) for the four GI-ReID and baseline rows is in app. 5: Baseline ResNet-50 75.81 / 97.34 / 98.95; GI-ReID ResNet-50 78.95 / 97.89 / 99.11; Baseline OSNet 83.68 / 98.24 / 99.26; GI-ReID OSNet 85.97 / 98.82 / 99.72 (Rank-1 / Rank-10 / Rank-20).

General and imitated-clothing video ReID app. 4, MARS with a ResNet-50 backbone. Columns are mAP and Rank-1.

CC-MARS is MARS with colour-jitter augmentation on the body region to imitate clothing change app. 4.

Other reported numbers. On the predicted sequences, GaitSet reaches 62.4% Rank-1 on CASIA-B §4.4. Feature maps for the image baseline concentrate on clothing, while GI-ReID activations also cover body structure and the face fig. 4.

Paper vs code

Repository: github.com/jinx-USTC/GI-ReID, linked from the paper's abstract. The first author is Xin Jin, so the owner belongs to an author. Commit a5179eaf6cb2b6cc575e2a45aa0c81a6330943ee (cloned at depth 1). The repository contains only the image-ReID baseline. The README says the gait prediction and regularisation code "will be released upon acceptance" README.md. Its code therefore cannot be used to check the GSP, GaitSet, SC or Phase 1–3 training.

What the release does contain, compared with the paper:

• Gait code absent. No GSP, GaitSet, SC layers, MMD or reconstruction loss exist in torchreid/. The gait arguments are defined but unused: lr_GCP 5e-4, stepsize_GCP 40, epochs-only-for-GCP 80, lr_GaitSet 5e-4, stepsize_GaitSet 120 args.py. The first three match the paper's Phase 2a values app. 2. lr_GaitSet differs from the paper's Phase 1 lr of 1e-4, and the paper states no GaitSet step at epoch 120. No released script reads them.
• Import error as shipped. train_Baseline.py imports FullTripletLoss and CFLLoss from torchreid.losses train_Baseline.py. The package __init__ exports only CrossEntropyLoss, TripletLoss and CenterLoss and defines DeepSupervision, so the two names are not defined there [code: torchreid/losses/init.py#L5-L7]. Read statically; not executed.
• Baseline schedule. The README's LTCC command uses 60 epochs, lr 3e-4, learning-rate steps at 20 and 40, batch 32 README.md. Defaults are Adam, weight decay 5e-4, margin 0.3 args.py. The paper's ReID-stream schedule is 240 epochs, lr 5e-4, weight decay 1e-5, steps at 80 and 160 app. 2. The paper does not state which schedule its reported baselines used.
• Batch sampling. The paper's Phase 3 samples P x N = 10 x 8 app. 2. The release uses the identity sampler only when --train-sampler RandomIdentitySampler is set data_manager.py; the default is an empty string args.py and the README commands do not set it.
• Triplet loss. Batch-hard mining matches the paper's description hard_mine_triplet_loss.py. The margin is 0.3 by default, while the paper states no ReID margin args.py.
• Loss weights. Baseline weights are 1.0 for classification and triplet args.py, matching the paper's Phase 3 values for those two terms.
• Evaluation. The baseline ranks by squared Euclidean distance on unnormalised features train_Baseline.py. The paper does not state feature normalisation for r.
• Cloth-changing metric. The rule that removes gallery images with the same identity and either the same camera or the same clothes eval_metrics.py matches the paper's caption for the cloth-changing setting tab. 6. It is active only when --use-cloth-changing-metric is passed, and the README warns to add it by hand README.md. The cuhk03 path has this rule commented out eval_metrics.py, and a comment in evaluate() says the cloth-ID path is unfinished eval_metrics.py.
• Masks. The baseline loop loads and zero-pads masks train_Baseline.py, but the baseline loss never uses them. The padding is zero-width with default mask size 64 x 64 args.py.
• Archive not read. The repository also commits GaitReID_CVPR2022-main (1).zip. Listing it was denied, so its contents are unknown and it may hold more code.
• Licence. No LICENSE file exists in the repository. Cite only, do not copy code.

No other discrepancies can be checked, because the gait code is not in the release.

Pitfalls

• Only the baseline is in the repository. Implement GSP, GaitSet, SC and the three training phases from the paper. The release's gait arguments do not match the paper and are not read by any script.
• Eq. 5 is moment matching, not kernel MMD. The paper calls it MMD, but the formula compares means and σ terms eq. 5. The text says "mean, variance" while the formula writes σ. Decide whether σ is the standard deviation or the variance; the two give different losses.
• Loss weights are not in the main text. Phase 3 weights (1.0, 0.1, 0.5) appear only in alg. 1 and app. 2.
• The prediction loss changes between phases. Phase 2 uses the squared-L2 loss of eq. 4 against the ground-truth sequence. Phase 3 has no ground truth and uses an L1 distance to the middle predicted frame app. 2.
• Frame indexing. The position label is p_mid = N // 2 §3.1. The supplement describes the middle frame as "the fourth one when N = 8" app. 2, i.e. 1-based index 4 (0-based index 3). The equation does not state the base.
• Phase 2 decay is written two ways. Warm-up decays at epoch 40 by a factor of 0.1, while joint training decays by 0.5 at 40, 80 and 120 app. 2. Both are as written; the paper does not explain the difference.
• Input geometry. Gait nets take 1 x 64 x 64 silhouettes. ReID masks are 2:1, so they are resized and zero-padded to a square, not stretched app. 1. The paper does not state the pad widths.
• Mask source. The paper names PointRend as an example segmenter for silhouettes §3. The extractor used for each dataset is not stated, and mask quality directly affects GSP input.
• Ablation without GSP duplicates one silhouette. The "GS (concat)" rows feed the single silhouette repeated as a set into GaitSet §4.2. Do not read them as GaitSet on real sequences.
• Inference feature choice. The paper uses r at test time. The aligned feature r̂ + ĝ scores 0.4 to 0.5 mAP higher on LTCC tab. 7, but it needs the gait stream, which is the cost the method removes.
• ReID triplet margin is not stated. The baseline default 0.3 args.py is not the paper's value; the paper states none.
• Sizes not stated. Embedding dimension for r̂ and ĝ is not stated §3.3. The release's reid-dim and gait-dim defaults are 256 args.py, but they belong to the baseline.
• Failure modes. Occlusion, partial bodies and multi-person images make GSP predictions unreliable §4.6 app. 6. The paper argues that the regularisation design limits sensitivity to these errors, but gives no quantitative test.
• Cost is hidden. Training runs GSP, GaitSet and the SC layers; test-time cost is the ReID backbone alone §3.3 §4.5. The paper reports no training time, memory or FLOP figures.
• Two evaluation settings. "Standard" discards same-identity, same-camera gallery images; "cloth-changing" also discards same-clothes images tab. 3 tab. 6. Do not compare the two.
• Column order differs between tables. Tables 3, 6 and 7 list mAP before Rank-1; Table 8 lists Rank-1 before mAP. Read each header before copying numbers.
• Real28 is test-only. It is evaluated with models trained on VC-Clothes §4.2 app. 3.
• LTCC-shape rows report only cloth-changing settings tab. 8a.
• PRCC table typo. "91,67" in the source is 91.67 here.
• Results are author-reported on specific splits; compare only within the same table and setting.

Lineage

• Builds on GaitSet for the gait recogniser and on set-based gait recognition §3.1. Also uses GSP's frame-interpolation ideas from the video-prediction literature §2.
• Compared against: GaitSet and PA-GCR as pure gait recognisers tab. 5; FITD (video motion cues) §2; Silhouette-ReID tab. 4; LTCC-shape, PRCC-contour and RCSANet as cloth-changing baselines tab. 8.
• Superseded by: not known from content already on this wiki.