Diverse Embedding Expansion Network and Low-Light Cross-Modality Benchmark for Visible-Infrared Person Re-identification

In plain words

Visible-infrared person re-identification asks whether a person seen by a daytime colour camera is the same person seen by a night-time infrared camera. The two kinds of picture look very different, and there are few paired examples to learn from. This paper's network creates extra versions of a person's internal features, which act as more training material and are pulled towards the other camera's view of the same person. A second module mixes features from several depths of the network, so that fine detail and broad context both reach the final description. The authors also collect a new dataset recorded in low light, where existing benchmarks are weaker, and report results for several methods on it.

TL;DR

• Two-stream ResNet-50 (separate visible and infrared stems, shared trunk) with a diverse embedding expansion (DEE) module that adds generated embeddings inside the network, trained with a centre-guided pair mining (CPM) loss and an orthogonality loss §3.1 §3.2 §3.3.
• A multistage feature aggregation (MFA) block adds channel-wise and spatial attention between stages §3.4.
• Author-reported gains on SYSU-MM01, RegDB and the new LLCM dataset. The LLCM data is not public: it needs a signed release agreement README.md.
• The official PyTorch code has no LICENSE file. It differs from the paper in several places (see Paper vs code), so do not treat the paper's equations as a complete recipe.

Problem

Cross-modality matching suffers from a large gap between visible and infrared images, and from few training pairs. Earlier feature-level methods struggle to project both modalities into one space directly, and image-translation methods add noise from missing paired data §1. Existing datasets are recorded under sufficient light, so they do not test the effect of strong illumination changes §1 §4.1.

Method

Architecture

A two-stream ResNet-50 is the backbone §3.1. Each modality has its own stem (the first convolution, normalisation, activation and max-pool), and the four residual stages are shared model.py. Features pass through MFA blocks after the early stages, and the DEE module sits inside the trunk §3.1 model.py. The paper does not define its stage numbering. In the code, the DEE module follows the third residual stage for SYSU-MM01 and LLCM, and the second stage for RegDB, where the fourth stage is removed §5.2 model.py.

During training, the features before and after the batch-normalisation layer are each fed to a loss §3.1.

Diverse embedding expansion (DEE)

Each DEE branch takes the feature map f and produces one generated embedding f+^i §3.2 eq. 1:

1. Three 3×3 convolutions with dilation 1, 2 and 3 each reduce the channels to one quarter §3.2.
2. Their outputs are summed and passed through a ReLU §3.2 eq. 1.
3. A 1×1 convolution restores the original channel count §3.2 eq. 1.

The generated embeddings are concatenated and fed to the next stage §3.2. The paper does not say along which axis they are concatenated.

# Diverse embedding expansion (DEE) branch  [§3.2] [eq. 1]
# features: (B, C, H, W) output of the stage the module follows
# returns: (B, C, H, W) generated embedding for this branch
def dee_branch(features):
    # three 3x3 convs with dilations 1, 2, 3, each to C/4 channels  [§3.2]
    maps = [conv3x3(features, dilation=d) for d in (1, 2, 3)]  # (B, C/4, H, W)
    # combine the three maps and apply ReLU  [§3.2] [eq. 1]
    hidden = relu(maps[0] + maps[1] + maps[2])
    # 1x1 conv back to C channels  [§3.2] [eq. 1]
    return conv1x1(hidden)  # (B, C, H, W)

# Expanded features passed to the next stage  [§3.2]
# num_branches: the paper's count of generated branches is not explicit here
def expand(features, num_branches):
    # one generated embedding per branch  [§3.2]
    generated = [dee_branch(features) for i in range(num_branches)]
    # all embeddings are concatenated; axis not stated  [§3.2]
    return concatenate([features] + generated, axis=0)

Centre-guided pair mining (CPM) loss

The paper states three properties that the generated embeddings should satisfy §3.3:

• Property 1: generated embeddings should differ from the original ones, so they carry new information §3.3.
• Property 2: a generated VIS embedding should move towards the original IR embedding of the same person, and vice versa §3.3.
• Property 3: intra-identity distance should stay below inter-identity distance, because Property 2 could otherwise pull identities together §3.3.

Eq. 2 states the loss on individual embeddings. The loss actually used works on per-identity centres. For a VIS-side generated embedding, the centre-form loss is §3.3 eq. 3:

# Centre-guided pair mining loss, VIS side  [§3.3] [eq. 3]
# centre_vis[j], centre_ir[j]: mean original embedding of identity j
# gen_vis[j]: mean generated VIS embedding of identity j (one branch)
# j, k: two different identities; how pairs are chosen is not stated [§3.3]
# dist: Euclidean distance  [§3.3]
def cpm_vis(centre_vis, centre_ir, gen_vis, margin=0.2):
    # pull the generated VIS centre to the IR centre, push it from the
    # original VIS centre, and keep identities apart  [eq. 3]
    return max(0, dist(centre_ir[j], gen_vis[j])
               - dist(centre_vis[j], gen_vis[j])
               - dist(centre_vis[j], centre_vis[k]) + margin)

The IR side swaps the modalities §3.3 eq. 4. The CPM loss is the sum of the two sides §3.3 eq. 5. The margin α is 0.2 §5.4.

Orthogonality between branches, eq. 6 §3.3:

# Orthogonality of generated embeddings  [§3.3] [eq. 6]
# branches: list of (B, d) generated embeddings, one per branch
def ort_loss(branches):
    total = 0
    # every pair of distinct branches m < n  [eq. 6]
    for m in range(len(branches)):
        for n in range(m + 1, len(branches)):
            # dot product per embedding; batch reduction not stated  [eq. 6]
            total += dot(branches[m], branches[n])
    return total

Multistage feature aggregation (MFA)

Each MFA block receives the features after a stage (f_h) and before it (f_l) §3.4. It has a channel branch and a spatial branch, both residual §3.4 eq. 7 eq. 8 eq. 9.

# Multistage feature aggregation, one block per stage  [§3.4] [eq. 7-9]
# f_high: features after the stage; f_low: features before the stage
def mfa_block(f_high, f_low):
    # channel branch: channel-to-channel similarity with softmax  [eq. 7]
    query = conv1x1_q(f_high)
    key = conv1x1_k(f_low)
    value = conv1x1_v(f_low)
    channel_sim = softmax(matmul(query, transpose(key)))
    # map attended channels back and add the input  [eq. 8]
    f_c = conv1x1_wc(matmul(value, channel_sim)) + f_high
    # spatial branch: the similarity M^s is named but never defined  [eq. 9]
    spatial_sim = spatial_similarity(f_c, f_low)  # definition not stated
    f_s = conv1x1_ws(matmul(conv1x1_v2(f_low), spatial_sim)) + f_c
    return f_s

Total objective

The four losses are summed with two weights λ1 and λ2 §3.5 eq. 10:

# Total objective  [§3.5] [eq. 10]
def total_loss(logits, labels, feats_pre_bn, feats_post_bn, cpm, ort):
    # cross-entropy on the classifier output  [§3.5]
    loss = cross_entropy(classifier(feats_post_bn), labels)
    # triplet loss on the pre-BN features; the assignment is not stated
    # in the paper, this follows the code  [§3.5]
    loss = loss + triplet(feats_pre_bn, labels)
    # weighted CPM and orthogonality terms  [§3.5] [eq. 10]
    loss = loss + lambda_1 * cpm + lambda_2 * ort
    return loss

Hyperparameters

Results

Author-reported. Setup: two-stream ResNet-50, SYSU-MM01 in all-search and indoor-search modes, RegDB in both VIS→IR and IR→VIS modes, LLCM in both directions. Baselines are the authors' selection; those marked * on LLCM were re-run with random erasing §5.1 §4.2 tab. 3. Only DEEN and the strongest baselines are reproduced here; the full comparison is in tab. 2.

SYSU-MM01 and RegDB tab. 2

The DEEN row has R-10 and R-20 values as well in tab. 2 (SYSU all: 97.6 and 99.2; RegDB VIS→IR: 97.8 and 98.9; RegDB IR→VIS: 96.8 and 98.4). The MMN, MAUM values for those columns are also in tab. 2. The paper's text says DEEN is best on every comparison §5.3; the table does not support that (see Pitfalls).

LLCM tab. 3: complete table.

Ablation: components tab. 4. Settings: DEE, L_cpm, L_ort, MFA.

Ablation: DEE stage tab. 5

Ablation: number of DEE branches tab. 6

Comparison with a non-local block tab. 7

Sensitivity to λ1, λ2 and α is shown only as a plot fig. 6. The text gives the best values as λ1 = 0.8, λ2 = 0.1 and α = 0.2 §5.4.

Paper vs code

Code: https://github.com/ZYK100/LLCM at commit 886e27caca0f5dbd7424df1ea81b4ad02f95cfb6. The DEEN implementation is in the DEEN/ folder. The repository is the authors'; the paper links it.

• Dilated branches are averaged, not summed. Eq. 1 sums the three dilated convolutions. The code divides their sum by three model.py model.py eq. 1.
• Generated embeddings are concatenated along the batch axis. The code stacks the original and generated features along dimension 0, tripling the batch model.py. The paper leaves the axis open §3.2.
• Dropout on the DEE output. The code applies dropout with p = 0.01 to the expanded features model.py model.py. The paper does not mention it.
• No softmax in the attention. Eq. 7 applies softmax to the channel similarity. The code scales the product by the channel count and applies no softmax model.py eq. 7. The spatial branch scales by the number of low-level positions, again without softmax model.py.
• Zero-initialised residual branches. The code initialises the output normalisation of each MFA branch to zero, so each block starts as the identity model.py model.py. The paper does not say this.
• Stride-2 projections in later blocks. In the second and third MFA blocks, the code uses stride 2 for the low-level projection and for the output projection of the channel branch model.py. The MFA blocks are placed after the stages listed in model.py. The paper does not describe the strides.
• CPM loss is not eq. 3–5. The code computes centres for four groups (VIS, IR, generated VIS, generated IR) and builds hinge terms from maximum and minimum distances over same-identity or different-identity masks loss.py. It adds three hinge terms per side with weights 1, 0.5 and 0.5 loss.py, and divides the result by two loss.py. The paper has one hinge per side eq. 3 eq. 4.
• Push-away term is detached. The same-identity distance that eq. 3 and eq. 4 maximise is computed with .detach() in the code loss.py loss.py. Gradients therefore do not push generated embeddings away from the original same-identity embedding, although the paper's Property 1 asks for this.
• Positive term uses the hardest same-identity pair. The code takes the maximum over same-identity distances loss.py loss.py. The paper does not say which same-identity pair is used.
• Orthogonality loss is on backbone features, not on the DEE output. Eq. 6 acts on the generated embeddings f+. The code takes the pooled backbone output of the two generated branches, computes their dot product per sample, and divides the sum by the full tripled batch size model.py. The weight also differs from the text: λ2 is 0.1 in §5.4 but 0.01 in the code default train.py.
• Learning-rate schedule. The paper decays to 1e-3 at epoch 60 and to 1e-4 at epoch 120 §5.2. The code decays to 1e-3 at epoch 80 train.py. Its 1e-4 step (elif epoch >= 120) comes after the epoch >= 80 branch, so it is never reached train.py. The code's backbone rate is 0.1× the head rate, which the paper does not state train.py.
• Test set used for checkpoint selection. The code evaluates on the test gallery after every epoch and keeps the checkpoint with the best Rank-1 train.py. The paper does not say how its reported checkpoint was chosen.
• Test-time fusion. The code sums six similarity matrices: the original, each generated branch, before and after BN train.py. The paper does not describe test-time scoring.
• Augmentation parameters. The code adds grey-scale, padding and crop steps, and uses different random-erasing ratios for SYSU-MM01 and for RegDB/LLCM train.py. The paper names only flips and erasing.
• Official results differ. The README replaces the LLCM results with updated numbers README.md. The DEEN row is 55.52 / 62.07 for one mode and 69.21 / 55.52 for the other README.md. The table has no mode labels, and the second mAP equals the first Rank-1, which looks like a copy error. Cite the paper's Table 3 as published and say the README differs.
• LLCM test cameras. The code's IR test list omits camera 3 data_manager.py data_manager.py. The paper states nine RGB/IR cameras §4.1.
• Training split files are not in the repository. The code reads idx/train_visible_<trial>.txt, and the default trial is 2 data_loader.py train.py. These split files are not in the repository, so the training split cannot be reproduced from the code.

Matches: the two-stream backbone with shared trunk, the DEE dilation values and channel reduction, the sampler (6 identities, 4 + 4 images), the input size, and the DEE placement per dataset all agree between paper and code.

Pitfalls

• Eq. 3–4 hinge structure. The paper's CPM is a single hinge per side. The code's version has three hinge terms with different weights and detached terms. Implement the code's form if you want to reproduce the reported numbers, and say so in your write-up.
• Orthogonality weight. The text says λ2 = 0.1 is best §5.4, while the code default is 0.01. The paper's settings line in the implementation section says 0.01, but it is commented out in the source. Check which value produced the reported numbers before you tune.
• Table 2 claim. The text says DEEN is best against all baselines §5.3. Table 2 does not support this: MMN has 91.6 Rank-1 for RegDB VIS→IR against DEEN's 91.1, MAUM has 84.3 mAP for RegDB IR→VIS against DEEN's 83.4, and VIS→IR mAP is tied at 85.1 tab. 2.
• Table 6 inconsistency. The "three branches" row (53.9 / 62.3 LLCM; 69.2 / 66.2 SYSU) matches no single row of Table 4. The LLCM numbers equal the DEE+CPM+L_ort row, while the SYSU numbers equal the DEE+CPM row tab. 4 tab. 6.
• LLCM split counts. Training has 30,921 boxes (16,946 VIS + 13,975 IR) and testing has 13,909 boxes, but the testing VIS and IR counts (8,680 + 7,166 = 15,846) do not sum to 13,909. The paper does not explain the gap §4.2. The 46,767 total also exceeds the train and test sums (44,830) by 1,937 boxes, which is not accounted for §4.2.
• Dataset access. LLCM requires a signed release agreement sent to the authors README.md. RegDB and SYSU-MM01 must be obtained from their own sites README.md README.md. The code expects ./Datasets/... paths train.py. Plan for data access before you plan the model.
• MFA spatial similarity is undefined. Eq. 9 uses M^s but the paper never gives its formula. The code's spatial branch is a different construction model.py.
• Stage numbering. The paper's stage-0 to stage-4 numbering is never defined, so "stage 3" is ambiguous. The code uses the stem output for MFA1 and places the DEE after layer3 for SYSU-MM01 and LLCM model.py.
• Where the pretraining comes from. The active text does not state it; the code loads ImageNet weights resnet.py. Confirm before comparing to other ImageNet-free results.
• Cost. The DEE output triples the batch seen by the later stages model.py. By reading the code, the compute and memory of those stages grow about threefold; this is an inference, not a measurement. The paper reports no run time or memory.
• Variance. The LLCM results average 10 random gallery splits §4.2. The code's test runs use one fixed split per run data_manager.py. The README warns the numbers vary with the split and that tuning may help README.md.
• Batch-hard triplet with pre-BN features. The triplet loss uses the pre-BN pooled features, and the classifier uses the post-BN features train.py. The paper only says both are used.

Lineage

• Builds on: ResNet-50 §3.1 [he2016deep]; the non-local block, adapted into the MFA block §3.4 [wang2018non]; the code is based on the cross-modal re-ID baseline by mangye16 README.md.
• Compared against: DDAG, LbA, AGW, CAJ, DART, MMN, MAUM, MCLNet, FMCNet and others tab. 2 tab. 3.
• Superseded by: not known from this wiki's content.