Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification

In plain words

Re-identification means recognising the same person across camera views. A model trained on one camera network often does worse on another, so people adapt it using unlabelled images from the new site. The usual recipe guesses a label for each unlabelled image by grouping similar images, then trains on those guesses. Some guesses are wrong, and wrong labels teach the model the wrong thing.

This method asks which guesses to trust. It runs two copies of the same model: a student that learns, and a slower-moving teacher that averages the student's past settings. For a given image, the two copies describe the image by how closely they match a set of reference people. If they agree, the label is probably right. If they disagree, the label is probably shaky. Shaky samples are given less weight in the training losses, so they pull the model less.

TL;DR

Clustering gives pseudo labels on the unlabelled target domain. For each target sample, compute the KL divergence between the student's and the mean teacher's soft class memberships over a reference set made of target cluster centres and source class centres. Use exp(-u) as a credibility weight in the ID classification, triplet and memory-bank contrastive losses, plus a penalty on the mean uncertainty. The reported gains are author-reported and come from ResNet-50 on DukeMTMC-reID, Market-1501 and MSMT17.

Problem

Clustering-based unsupervised domain adaptation for person re-identification alternates between pseudo-label generation on target images and feature fine-tuning on those labels §1. Label noise from clustering and from the domain gap misleads the fine-tuning §1. Earlier fixes either train two or more networks for mutual learning, or select reliable samples with a hard threshold that discards the rest §2.1. The authors aim for a single network with a soft, per-sample weight §2.1 §3. The method assumes that a wrong pseudo label goes with a larger disagreement between the two models. This is shown on one dataset pair, Duke to Market, as a plot of two distributions fig. 1 §1; no statistic is given.

Method

Pipeline. Three stages, as in other clustering-based methods §3.1:

1. Pre-train a ResNet-50 on labelled source data §3.1 §4.2.
2. Cluster the unlabelled target features, taken from the mean teacher, and assign pseudo labels to the clusters §3.1 fig. 2.
3. Jointly fine-tune. Source images use an ID classification loss, with the source class centres as the classifier weight vectors §3.1. Target images use the ID, triplet and contrastive losses against the pseudo labels, each weighted by uncertainty §3.2 fig. 2. Stages 2 and 3 alternate fig. 2.

Mean teacher. The teacher's weights are a moving average of the student's weights. No gradient flows into the teacher. Its features drive the clustering and the final ReID inference §3.1. The paper does not give the averaging factor in the published text §3.1 §4.2.

Memory bank. Each target query is contrasted against a bank of N target instances, maintained as in cross-batch memory §3.1. The bank also holds the C_s source class centres as extra negatives §3.1. Positives and negatives are defined by the pseudo labels §3.1. The paper uses a self-paced weight from circle loss, but the published text omits its form §3.2.

Reference set and uncertainty. Let R = [R_t, R_s] stack the K_t target cluster centres and K_s source class centres, with K_r = K_t + K_s rows §3.2. Source centres are the weights of the source ID classifier's fully connected layer §3.2. For a target feature f_i from the student, the soft multilabel is p_i. The same computation on the teacher feature gives p̃_i §3.2 eq. 1. The uncertainty is the KL divergence from the student's distribution to the teacher's, or the reverse, as the text states it eq. 2. The credibility weight is ω_i = exp(-u_i) §3.2.

# Soft multilabel of one feature over the reference set  [§3.2] [eq. 1]
# Input:  feature (D,); references R (K_r, D) = [target centres; source centres]
# Output: probs (K_r,)

def soft_multilabel(feature, references):
    # Similarity of the feature to every reference  [§3.2] [eq. 1]
    scores = references @ feature                 # (K_r,)
    # Normalise the similarities into a distribution  [§3.2] [eq. 1]
    return softmax(scores)

# Uncertainty and credibility of one target sample  [§3.2] [eq. 2]
# Input:  student_feat (D,), teacher_feat (D,), references R (K_r, D)
# Output: u (scalar uncertainty), w (scalar credibility)

def uncertainty(student_feat, teacher_feat, references):
    # Soft multilabel from the student model  [§3.2] [eq. 1]
    p = soft_multilabel(student_feat, references)           # (K_r,)
    # Soft multilabel from the mean teacher model  [§3.2] [eq. 1]
    p_tilde = soft_multilabel(teacher_feat, references)     # (K_r,)
    # KL divergence over the reference set  [§3.2] [eq. 2]
    u = sum(p_tilde * log(p_tilde / p))
    # Credibility weight, exp(-u), used in the losses  [§3.2]
    w = exp(-u)
    return u, w

Uncertainty-weighted ID loss. Each target sample's cross-entropy on its pseudo label is multiplied by its credibility and averaged over the mini-batch §3.2 eq. 3.

# Uncertainty-weighted ID loss on target samples  [§3.2] [eq. 3]
# Input:  p_label (n_t,): probability of each sample's pseudo label
#         w (n_t,): credibility weights
# Output: scalar

def weighted_id_loss(p_label, w):
    # Per-sample negative log-likelihood of the pseudo label  [eq. 3]
    nll = -log(p_label)                           # (n_t,)
    # Average of credibility-scaled terms over the batch  [eq. 3]
    return mean(w * nll)

Uncertainty-weighted triplet loss. The weight of an anchor-positive or anchor-negative pair is φ(u_a, u_p) = ω_a + ω_p §3.2 eq. 4. The loss is a softmax-style pair loss over the weighted positive and negative scores, with no margin in the published formula §3.2 eq. 5.

# Uncertainty-weighted triplet loss  [§3.2] [eq. 4] [eq. 5]
# Input:  s_ap, s_an (n_tr,): similarities of positive and negative pairs
#         w_ap, w_an (n_tr,): pair credibilities from eq. 4
# Output: scalar

def weighted_triplet_loss(s_ap, s_an, w_ap, w_an):
    # Credibility-scaled exponentials of each pair  [eq. 5]
    pos = w_ap * exp(s_ap)                        # (n_tr,)
    neg = w_an * exp(s_an)                        # (n_tr,)
    # Negative log of the positive share  [eq. 5]
    return mean(-log(pos / (pos + neg)))

Uncertainty-weighted contrastive loss. For a query, the bank's positives and negatives are each weighted by pair credibility before the exponentials are summed §3.2 eq. 6.

# Uncertainty-weighted contrastive loss over the memory bank  [§3.2] [eq. 6]
# Input:  s_pos, s_neg (n_t, N_pos), (n_t, N_neg): query similarities
#         w_pos, w_neg: pair credibilities of the same shapes
# Output: scalar

def weighted_contrastive_loss(s_pos, s_neg, w_pos, w_neg):
    # Weighted sum of exp(s) over the negatives  [eq. 6]
    neg_term = sum(w_neg * exp(s_neg), axis=1)    # (n_t,)
    # Weighted sum of exp(-s) over the positives  [eq. 6]
    pos_term = sum(w_pos * exp(-s_pos), axis=1)   # (n_t,)
    # Log of one plus the product, averaged over queries  [eq. 6]
    return mean(log(1 + neg_term * pos_term))

Total target loss. The target loss adds the three weighted losses and a penalty on the mean uncertainty, L_reg = mean(u_i) §3.2 eq. 7.

# Total fine-tuning loss on target data  [§3.2] [eq. 7]
# Inputs: the three weighted losses, per-sample uncertainties u (n_t,),
#         and the weights lambda_tri, lambda_ct, lambda_reg
# Output: scalar

def target_loss(l_uid, l_utri, l_uct, u,
                lambda_tri, lambda_ct, lambda_reg):
    # Regulariser that stops the model from making every u large  [§3.2]
    l_reg = mean(u)
    # Weighted sum of the four terms  [eq. 7]
    return l_uid + lambda_tri * l_utri + lambda_ct * l_uct + lambda_reg * l_reg

Hyperparameters

Results

Author-reported. Setup: ResNet-50 backbone, one network trained with a teacher copy §4.3. Protocol: source to target transfer on DukeMTMC-reID (Duke), Market-1501 (Market) and MSMT17 §4.1. Metrics: mAP and CMC Rank-1, 5 and 10 §4.1. All rows below are transcribed from tab. 1 of the paper, and only the rows the paper lists in the published text are included.

Headline: Duke to Market mAP 78.1 and R1 91.9; Market to Duke mAP 69.1 and R1 82.0. The paper reports margins over the second-best method of 2.1, 3.0, 2.4 and 2.9 mAP points for Duke to Market, Market to Duke, Market to MSMT17 and Duke to MSMT17 §4.3.

Duke and Market, tab. 1 (source underlined in the paper: MEB-Net for the first four columns of each direction, MMT for R5 and R10 of Market to Duke)

MSMT17, tab. 1 (the paper's column header reads "Marke1501→MSMT17"; this is a typo for Market-1501. Second best, underlined in the paper: MMT)

Component ablation, tab. 2 (Market and Duke; UNRN starts from SBase.)

Uncertainty design and L_reg, tab. 3.

Memory bank size, tab. 4 (Market→Duke, on top of Baseline + Source + Contrastive; the column "0" is labelled that way in the paper and is not defined further.)

Paper vs code

No official code. The paper has no code link [arXiv abstract page lists no code or project link]. I searched GitHub for "UNRN" (hits are unrelated projects of a university in Argentina), "noise resilient reid", "uncertainty reid mean teacher" and "domain adaptive person reid uncertainty", and searched GitHub users for the first author. None returned a repository tied to the authors.

Pitfalls

• Eq. 2 index typo. The sum runs over k, but the denominator in the published formula is p_{i,n}. Read it as p_{i,k} eq. 2.
• Pair weight. The text calls φ(u_a, u_p) "the average" of the two credibilities, but the formula is the sum ω_a + ω_p §3.2 eq. 4. The sum lies in (0, 2] for ω in (0, 1], while an average lies in (0, 1]. This changes the effective scale of L_UTRI and L_UCT. Implement the formula as written.
• Negative similarity typo. In the text after eq. 5, s_an is described as the similarity of the positive pair. It is the negative pair's similarity §3.2.
• Similarity measure is not defined. The published text uses s for similarities in eq. 5 and eq. 6 but never says which one (cosine, dot product, or other), and it does not state feature normalisation. Both change the exp() ranges and the uncertainties.
• Overflow. eq. 5 and eq. 6 take exp() of raw similarities, and eq. 1 takes softmax over unnormalised dot products. Compute in log space (log-sum-exp) to avoid overflow. The paper does not mention this.
• Self-paced weight missing. The text says the contrastive loss uses the circle-loss self-paced weight, but eq. 6 omits it and gives neither its form nor its scale §3.2.
• Gradient flow is not stated. Whether the teacher-side distribution is detached, and whether the gradient flows through ω_i into the features, is not stated. The KL term and the weights could each carry gradients.
• Source ID loss is not in eq. 7. The source batch's ID loss is described in words (source class centres as the classifier weights §3.1), with no equation and no weight. The ablation credits source data with about 2 mAP points on Duke to Market tab. 2. Rebuild that loss from the description and mark it as inferred.
• Triplet mining is not described. The published text does not say how positives and negatives are chosen for each anchor, or whether a margin is used. eq. 5 has no margin.
• Clustering details. The clustering uses teacher features §3.1, and the DBSCAN parameters are given §4.2. The distance metric, the re-clustering interval, and whether cluster centres come from the teacher or the student are not stated.
• Memory bank. N for the main results is not stated. The text refers to [wang2020cross] for how the bank is maintained §3.1. Table 4's 8192 row matches SBase. on Market to Duke, so N = 8192 is a likely reading, but it is an inference.
• Table 4 baseline mismatch. The caption of tab. 4 names the baseline "Baseline + Source + Contrastive loss". Its "0" column (62.8 mAP) does not match the "+ Contrastive loss" row of tab. 2 (62.1 mAP) on the same pair. The reason is not stated.
• Evidence for the core claim. The claim that wrong pseudo labels have higher uncertainty rests on one qualitative figure for Duke to Market fig. 1 §1. No correlation or AUROC is reported in the published text.
• Costs. Training uses two models (student and teacher) §4.3. The memory bank stores N feature vectors §3.1.
• Averaging factor. The EMA momentum is not given in the published text. Set it explicitly and record the choice.
• Typos and a wrong citation key. The tab. 1 header "Marke1501→MSMT17" is a typo for Market-1501. In the comparison text, MEB-Net is cited with the key Zhai2020, which the bibliography resolves to AD-Cluster (CVPR'20). MEB-Net is the separate entry zhai2020multiple (ECCV'20) §4.3. Check the reference before relying on it.

Lineage

• Builds on: mean teacher, which averages student weights over training (Tarvainen and Valpola, 2017); cross-batch memory for embeddings (Wang et al., CVPR 2020); self-paced weighting from circle loss (Sun et al., CVPR 2020); the uncertainty weighting policy exp(-u) from Kendall and Gal (NeurIPS 2017) §3.2; the soft-multilabel idea over reference persons from MAR (Yu et al., CVPR 2019) §3.2.
• Compared against: MMT, a mutual mean-teaching method with four models (Ge et al., ICLR 2020) §4.3; MEB-Net, with six models §4.3; SSG, NRMT, DAAM, AD-Cluster and others in tab. 1.
• Superseded by: not stated in the published text.