Beyond Appearance: a Semantic Controllable Self-Supervised Learning Framework for Human-Centric Visual Tasks

In plain words

Most self-supervised pretraining on pictures of people learns how people look: clothes, colours, the overall appearance that tells one person from another. That helps with re-identification, but tasks such as parsing a body into parts or finding pedestrians need to know what each region is, and that information is weak in those features.

SOLIDER adds a self-made semantic signal. It groups the image patches into a few body regions, labels them, and asks the network to predict those labels, including when one region has been hidden. The labels come from the images themselves, so no human annotation is needed. Training uses about four million unlabelled person photos.

It also adds a dial. A single number fed into a small controller changes how much the output features emphasise appearance versus body-part meaning. A re-identification system can turn the dial one way and a parsing system the other, using the same pretrained weights.

TL;DR

• Pretrains a Swin backbone on unlabelled person images (LUPerson, 4.18M images) §4.1.
• Adds a token-level classification loss on pseudo part labels from K-means over teacher features, with one randomly masked part per image §3.1.
• A semantic controller takes a value λ and modulates each Swin stage, so the same weights give different appearance/semantic mixes §3.2.
• Author-reported gains over DINO and over prior baselines on most metrics; the authors note that PedesFormer, which uses extra data, is better on pedestrian detection §4.4.
• Relevant if you need a person-centric pretrained backbone, or want one model that can be tuned toward different semantic levels.

Problem

Self-supervised pretraining such as DINO builds features that group patches by visual appearance, so different body parts of one person end up close together §1 fig. 1. The authors argue that human-centric tasks need more semantic information, and that different tasks need different ratios of semantic and appearance information, which a single fixed representation cannot supply §1 §3.2. Earlier work that clustered DINO features for semantic labels was dominated by appearance §3.1.

Method

SOLIDER starts from DINO and adds two parts: a semantic supervision loss built from pseudo labels §3.1, and a controller that makes the network conditional on λ §3.2. The DINO backbone is Swin-Transformer §4.1.

Pseudo semantic labels §3.1

1. Run the teacher on the two global views and take its last-stage token features, which the paper uses for clustering §3.1 fn. 4.
2. Split the tokens of each image into foreground and background by 2-means clustering on the L2 norm of each token. The cluster with the larger norm is foreground §3.1.
3. Cluster the foreground tokens into N parts with k-means, where N = 3 in the paper's own setup §3.1.
4. Label the parts by vertical position, with rows counted from the top: the part with the smallest mean row index (highest in the image) gets label 1. The paper names the parts upper body (heads, chests, arms), lower body (waist, thighs) and shoes (calves, feet) §3.1 §3.1 fn. 3. The background gets label 0, so there are N+1 classes eq. 1.
5. Hide one of the N parts in the image, then feed the masked image through the student as well §3.1.

# SOLIDER training step  [§3.1] [§3.2] [eq. 1] [eq. 3]
# Input:  global_views (B, 3, H, W), local_views (list of crops)
# Output: scalar loss

def training_step(global_views, local_views):
    # Teacher pass with lambda = 1 gives the clustering features
    # [§3.1 fn. 4] [code: main_solider.py#L375-L377]
    teacher_feats = teacher(global_views, lambdas_ones)  # (2B, C, h, w)

    # Teacher pass with a random binary lambda per image gives DINO targets
    # [§3.2] [code: main_solider.py#L380-L382]
    lambdas = sample_binary(p=0.5, size=B)  # values in {0, 1}
    teacher_out = teacher(global_views, lambdas)

    # Pseudo part labels: foreground by norm, then N parts, sorted by y
    # [§3.1] [code: utils.py#L658-L700]
    labels, valid = pseudo_part_labels(teacher_feats, num_parts=3)

    # Hide one randomly chosen part; the same part for the whole batch
    # [§3.1] [code: main_solider.py#L388-L394]
    part = random_int(1, num_parts)
    masked_views = global_views[valid] * (labels[valid] != part)

    # Student sees masked and unmasked views, each with its own lambda
    # [§3.1] [code: main_solider.py#L397-L399]
    student_out, student_feats = student(masked_views + all_crops, lambdas)

    # Per-token cross-entropy on masked and unmasked global views
    # [eq. 1] [code: main_solider.py#L408-L413]
    semantic_logits = part_head(student_feats)  # (tokens, N+1)
    semantic_loss = cross_entropy(semantic_logits, labels)

    # Semantic term only counts where lambda = 1 [eq. 3]
    # [code: main_solider.py#L420-L425]
    semantic_loss = lambdas * semantic_loss  # lambda broadcast to tokens

    # DINO cross-entropy over the student crops [§3.1]
    # [code: main_solider.py#L417]
    dino = dino_loss(student_out[num_global:], teacher_out)

    # Paper weights the terms by alpha = 0.5; code uses 1:1 [eq. 2]
    # [code: main_solider.py#L426]
    return dino + semantic_loss

The paper's semantic term is the token-level cross-entropy in eq. 1, and the total loss mixes it with the DINO loss through a balance weight alpha eq. 2. The controller version uses λ to scale the semantic term eq. 3.

# Semantic controller, applied after each Swin stage [§3.2]
# Input:  x (B, L, C) stage output; lam (B,) in [0, 1]
# Output: modulated x (B, L, C)

def apply_controller(x, lam, stage):
    # Encode lambda as the pair [lam, 1 - lam]
    # [code: swin_transformer.py#L1338]
    code = stack([lam, 1 - lam], axis=-1)  # (B, 2)

    # Map the code to a per-channel gain and bias
    # [code: swin_transformer.py#L1350-L1351]
    gain = softplus(linear_w[stage](code))  # (B, C), non-negative
    bias = linear_b[stage](code)  # (B, C)

    # Multiply and shift every token of the stage output
    # [§3.2] [code: swin_transformer.py#L1352]
    return x * gain[:, None, :] + bias[:, None, :]

Semantic controller §3.2

The controller takes λ in [0, 1] and encodes it as a weight vector and a bias vector. After a Softplus, the weight vector multiplies the feature maps and the bias vector is added §3.2. The paper describes it as applied after each Swin block §3.2; the code applies it after each of the four Swin stages swin_transformer.py. At pretraining λ is sampled from a binary distribution, Bernoulli(0.5) over {0, 1}. The authors also tried uniform U[0,1] and Beta(0.2, 0.2), which both did worse §3.2. At downstream time λ is set by hand §3.2.

Two-stage schedule §4.1

The authors first train DINO for 100 epochs, then fine-tune SOLIDER from that model for 10 more epochs at a smaller learning rate, because the clustering step is slow §4.1. The released code does this with resume_solider.sh resume_solider.sh.

Hyperparameters

Paper values are cited to the paper. Values marked (code) are read from the official repository and are not in the paper.

Results

Author-reported. Setup: LUPerson pretraining; downstream baselines are reproduced by replacing their backbone with the pretrained Swin §4.1. Table 1 is the ablation on Swin-Tiny with the paper's downstream protocols §4.3.

Source: tab. 1. The text notes that the re-ID and person-search numbers dipped after adding clustering alone and recovered with the controller §4.3.

Table 2 compares against prior methods on the same benchmarks tab. 2. Only the highest prior per row is shown here; the full rows are in the paper.

Source: tab. 2 §4.4. The text says PRW mAP is more than 10 points above prior work §4.4. Pedestrian detection is better with PedesFormer, which the authors say uses extra autonomous-driving data §4.4. The text also says the pose result (76.6/81.5) is above the MMPose figure for HRFormer with a Swin backbone (75.9/81.1) §4.4. The column headed "Swin-T" has no "SOLIDER" label in the paper; the repository README lists the same numbers as SOLIDER Swin Tiny models README.md.

Paper vs code

The official repository is linked from the abstract. It is at commit b8779cc5cfaa7d44b277e8305bc42bce377fe8c0 and has an Apache-2.0 LICENSE. Some files keep the copyright header of the DINO code this repo builds on.

• Optimizer. Paper: SGD §4.1. Code: AdamW is the default, and the run script does not change it main_solider.py main_solider.py.
• Learning rate. Paper: 0.0005 §4.1. Code: 0.0005 is the reference rate and is scaled by batch size: 0.0005 × 384 / 256 = 0.00075 for 8 GPUs at 48 per GPU main_solider.py main_solider.py. The paper's warmup is not stated; the code uses 10 epochs main_solider.py.
• Loss weights. Paper: α = 0.5 on DINO and (1 − α) on the semantic term eq. 2. Code: 1 on DINO and semantic_loss = 1.0 on the semantic term, with no 0.5 factor main_solider.py main_solider.py. The ratio is the same as α = 0.5, but the total is twice as large. That matters if you switch the optimizer to SGD, as the paper says.
• λ per sample or per iteration. Paper: λ is sampled once per iteration §3.2. Code: a separate binary λ is drawn for each image in the batch, and the same vector is used by teacher and student main_solider.py main_solider.py.
• Where the controller sits. Paper: after each block §3.2. Code: after each Swin stage swin_transformer.py.
• Controller on the last stage. The code computes the stage-3 gain and bias, but the output features are taken from the tensor before this modulation. So the last-stage controller does not change the features that are returned or clustered swin_transformer.py swin_transformer.py. The paper does not discuss this.
• DINO baseline. The DINO run uses the same Swin class, and its forward is called without λ, so the controller is present with λ fixed at 0 main_dino.py swin_transformer.py swin_transformer.py. The paper does not say whether its DINO baseline has the controller.
• Semantic head. Paper: blocks of FC, batch norm and ReLU §3.1 fn. 5. Code: 3-layer GELU MLP with no batch norm, L2-normalised output, and a last layer whose weight norm is frozen at 1, so logits are cosines in [−1, 1] vision_transformer.py main_solider.py.
• Masking. The paper says the masked image is re-fed §3.1. The code multiplies the normalised image by a mask, so masked pixels become 0 in normalised space, which is the dataset mean colour main_solider.py. The mask is upsampled from the token grid with nearest-neighbour interpolation main_solider.py. One part index is drawn per batch, not per image main_solider.py.
• Second stage. The paper says SOLIDER is fine-tuned from DINO. The code restores student, teacher and the fp16 scaler, but not the optimizer, the DINO head centre, the semantic head, or the epoch counter, so the schedule restarts at epoch 0 main_solider.py utils.py.
• Masked and unmasked views. The semantic loss is applied to both views §3.1. The code computes it over the masked input and the two unmasked global views for images that pass the foreground test main_solider.py.
• Missing details. The paper's pseudocode is in its supplementary material, which is not in the arXiv source. The paper's input size, crop scales, and output dimension are not stated; the code uses the values in the table above.

Pitfalls

• Dead controller at the last stage. The code's stage-3 controller parameters receive no gradient, and the training script relies on DDP's find_unused_parameters=True main_solider.py main_solider.py. If you port the design, decide whether the last-stage modulation should reach the output.
• Bounded semantic logits. With frozen last-layer gain and cosine logits, the semantic cross-entropy has a floor. For 4 classes (N = 3 plus background), the smallest possible loss is ln(1 + 3e⁻²) ≈ 0.34 nats. This is our own calculation from the code, not something the paper reports. Check the loss curve against this floor before reading it as a training signal.
• Paper's SGD is untested in the code. The released code runs AdamW. If you follow the paper's SGD, the 2× difference in total loss scale matters, and the learning rate may need retuning main_solider.py.
• Clustering cost. Each step runs the teacher twice and fits two K-means models per image on the CPU main_solider.py utils.py. The paper calls the clustering time-consuming §4.1. The code runs it in the training loop, not as a precomputed step.
• Images without a usable foreground are skipped. If foreground covers at most half the tokens, that image gets no semantic label in that step utils.py. The paper does not state this threshold.
• Token grid. The paper gives no input size. With the run script's 256×128 global crops and a stride of 32 in the last Swin stage, the token grid is 8×4 run_solider.sh. This is inferred from the code.
• Downstream weights. init_weights loads the teacher key when it is present swin_transformer.py. Use the same key when you load a checkpoint downstream. It loads with strict=False, so missing keys are silently skipped swin_transformer.py.
• Averages in the text. The ablation text says DINO on LUP1M gives "an average of 1.4 improvement (72.6 vs. 74.0)" over ImageNet supervision, and that SOLIDER reaches 74.9 §4.3. The first comparison is written in the wrong order, and the averaging rule is not stated, so the averages do not map clearly to the table columns.
• Downstream λ. Only ranges are given in the paper §4.3. The exact value per task is in the supplementary material, which is not in the source.

Lineage

• Builds on: DINO, the baseline that SOLIDER starts from §3.1, and Swin Transformer, the backbone §4.1.
• Related: MAE is named as sharing DINO's appearance-focused behaviour §1 fn. 1.
• Compared against: TransReID and PASS (both trained on LUPerson), attribute and person-search methods, and PedesFormer tab. 2 §4.4. These papers have no card here and no arXiv id in the source bibliography, so they are named only.
• Superseded by: not known from content in this wiki.