Paper · 2022-07-16 · cs.CV
Learning Granularity-Unified Representations for Text-to-Image Person Re-identification
LGUR maps person images and text separately into K=6 part vectors in one granularity-unified space, using a shared 400-atom dictionary and learned prototype queries, so testing needs no cross-modal attention. Author-reported Rank-1 on CUHK-PEDES: 64.21 (ResNet-50) and 65.25 (DeiT-Small). On ICFG-PEDES: 57.42 and 59.02.
Cross-modal retrievalMetric learningHard negative miningObject queries (learned query slots in transformer decoders)text-to-image-reidgranularity-gapdictionary-reconstructionlearned-prototypes
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Finding a person from a written description is hard partly because the picture holds far more detail than the sentence. "Black and white jacket" fits many photos, and the photos differ in the things the words leave out. This paper targets that mismatch in detail level, which it calls a granularity gap, and not only the gap between words and pixels.
The model turns each photo and each sentence into a small set of part-level vectors that live in one shared space. A learned dictionary of common building blocks sits between the two sides, and a few learned query slots pick out the parts. Photos and sentences are encoded separately, so a search can compare stored vectors directly, without pairing every query with every photo.
Training uses helper signals that are dropped at search time: a mask that focuses on the person rather than the background, and a text-guided copy of the image features that pulls the two sides closer. The authors report better accuracy than earlier methods on two public benchmarks, and their numbers are their own.
TL;DR
- Text-to-image person re-identification retrieves pedestrian images from a natural-language description. §1
- LGUR encodes images and text separately, reconstructs both through a shared learned dictionary, and then maps them to K=6 part vectors. §3
- Training uses an identification loss and a ranking loss, plus a guidance loss that is dropped at inference. §3.4
- Retrieval uses cosine similarity between reconstructed features only, so no image-text pair has to be fed through the network together. §3.4 §2.2
- Author-reported: 65.25 Rank-1 on CUHK-PEDES with DeiT-Small, 59.02 on ICFG-PEDES. tab. 1 tab. 2
Problem
Text-to-image person re-identification searches a gallery of pedestrian images by a text query §1. Visual features are fine-grained while text is coarse, so one description can match several images whose visual details differ §1 fig. 1. Earlier cross-modal attention methods align words with image regions, which handles varying patch sizes but not this difference in detail level §2.2. Those methods also need every image-text pair to be fed through the network to score a match, so their cost grows with the product of the number of images and sentences, O(MN) §2.2.
Attention-free methods are cheaper, O(M+N), but usually retrieve less well §2.2. The paper targets the granularity gap while keeping that cheap test-time cost §1.
Method
LGUR has three parts: backbones that give visual and textual features (§3.1), a Dictionary-based Granularity Alignment module (DGA, §3.2), and a Prototype-based Granularity Unification module (PGU, §3.3). Optimisation and inference are in §3.4 fig. 2.
Backbones §3.1. Visual features are patch tokens from DeiT-Small with patch size 16, giving HW tokens of width d §3.1 §4.2. Textual features come from BERT, which is frozen, followed by a bidirectional LSTM §3.1 §4.2. The paper also describes a ResNet-50 visual backbone §3.1; the code's training entry point builds only the DeiT-Small path (see Paper vs code).
DGA §3.2. A Multi-modality Shared Dictionary D (s atoms of width d) is randomly initialised §3.2. Text and image tokens each query D with a transformer block MHA_1, whose keys and values are D eq. 1 eq. 2. The paper states MHA_1 as multi-head attention plus a feed-forward network, without residual connections or normalisation §3.2.
Two extra terms shape D during training only §3.2:
- Foreground mask. A per-token sigmoid gate M, from a 1x1 convolution on the visual tokens, multiplies the reconstructed visual features §3.2 eq. 2.
- Cross-modal guidance. The visual tokens are reconstructed again, now attending to the text tokens, and gated by M eq. 3.
PGU §3.3. A set P of K learned prototypes, randomly initialised and shared by both modalities, serves as queries §3.3. Each prototype attends to one feature map through MHA_2 (same structure as MHA_1, separate weights), then an FC layer W_k specific to that prototype maps the result to d' eq. 4. The FC layers are per prototype but shared by the two modalities §3.3. The same PGU is applied to four feature maps: reconstructed text, reconstructed visual, plain text, and guided visual eq. 5. The paper states that a single shared MHA_2 is used in eq. 5 §3.3.
Optimisation §3.4. Each PGU output gets an identification loss, the mean over K prototypes of a cross-entropy against the identity label eq. 6. Matched pairs get a cosine ranking loss with margin alpha and one semi-hard negative per direction eq. 7. The total loss is the sum of the four identification terms, two ranking terms between reconstructed and cross-modal features, and two guidance terms that tie reconstructed features to their guidance features §3.4. All loss weights are unit weights §3.4. At inference only the reconstructed text and visual features are used, and cosine similarity ranks the gallery §3.4.
Equations as written in the source:
T_{re} = MHA_1(T, D, D) eq. 1
V_{re} = MHA_1(V, D, D) \odot M eq. 2
V_{g} = MHA_1(V, T, T) \odot M eq. 3
\widetilde{F} = PGU(P, F) = Concat(f_1(p_1, F), \dots, f_K(p_K, F)) eq. 4
f_i(p_i, F) = W_k(MHA_2(p_i, F, F)) eq. 4
\widetilde{T}_{re} = PGU(P, T_{re}), \quad \widetilde{V}_{re} = PGU(P, V_{re}), \quad \widetilde{T} = PGU(P, T), \quad \widetilde{V}_{g} = PGU(P, V_{g}) eq. 5
L_{ID}(\widetilde{F}) = \frac{1}{K} \sum_{k=1}^{K} -y \odot \log(\hat{y}_k) eq. 6
L_{RK}(\widetilde{F}_1, \widetilde{F}_2) = \max(\alpha - S(\widetilde{F}_1, \widetilde{F}_2^{+}) + S(\widetilde{F}_1, \widetilde{F}_2^{-}), 0) + \max(\alpha - S(\widetilde{F}_2, \widetilde{F}_1^{+}) + S(\widetilde{F}_2, \widetilde{F}_1^{-}), 0) eq. 7
The paper writes the overall objective in three unlabelled equations, cited here by section:
L_M = L_{ID}(\widetilde{T}_{re}) + L_{ID}(\widetilde{V}_{re}) + L_{ID}(\widetilde{T}) + L_{ID}(\widetilde{V}_{g}) + L_{RK}(\widetilde{T}_{re}, \widetilde{V}_{re}) + L_{RK}(\widetilde{T}, \widetilde{V}_{g}) §3.4
L_G = L_{RK}(\widetilde{T}_{re}, \widetilde{T}) + L_{RK}(\widetilde{V}_{re}, \widetilde{V}_{g}) §3.4
L = L_M + L_G §3.4
Pseudocode for the forward pass, the loss and retrieval. The code's batch-level ranking loss differs; see Paper vs code.
# Forward pass for one batch [§3]
# images: (B, 3, 384, 128); caption_ids, caption_mask: (B, L)
def encode(images, caption_ids, caption_mask, dictionary, prototypes):
# DeiT-Small patch tokens, class token removed
# [§3.1] [code: model/model.py#L174-L177]
visual = deit_small_patch_tokens(images) # (B, HW, d)
# Frozen BERT, then Bi-LSTM; forward and backward states averaged
# [§3.1] [code: model/text_feature_extract.py#L76-L112]
textual = bilstm(frozen_bert(caption_ids, caption_mask)) # (B, L, d)
# Text reconstruction through the shared dictionary [§3.2] [eq. 1]
text_recon = mha1(textual, dictionary, dictionary) # (B, L, d)
# Foreground mask: sigmoid of a per-token linear map [§3.2] [eq. 2]
mask = sigmoid(linear(visual)) # (B, HW, 1)
# Visual reconstruction, gated by the mask [§3.2] [eq. 2]
visual_recon = mha1(visual, dictionary, dictionary) * mask
# Guidance: visual tokens attend to text, gated by the mask
# (training only) [§3.2] [eq. 3]
visual_guided = mha1(visual, textual, textual) * mask
# Prototype pooling of all four feature maps [§3.3] [eq. 5]
text_recon_p = pgu(prototypes, text_recon)
visual_recon_p = pgu(prototypes, visual_recon)
text_p = pgu(prototypes, textual)
visual_guided_p = pgu(prototypes, visual_guided)
return text_recon_p, visual_recon_p, text_p, visual_guided_p
# Prototype unification: each prototype attends to one feature map
# [§3.3] [eq. 4]
def pgu(prototypes, features):
# One shared attention block for all calls [eq. 5] [§3.3]
attended = mha2(prototypes, features, features) # (B, K, d)
# Per-prototype FC layer, shared across modalities [§3.3] [eq. 4]
return stack([fc[k](attended[:, k]) for k in range(K)]) # (B, K, d')# Training objective, computed on the four PGU outputs [§3.4]
# Identity labels are used only here; mask and guidance are training-only
def training_loss(text_recon_p, visual_recon_p, text_p, visual_guided_p, ids):
# Identification loss on the four feature groups [§3.4] [eq. 6]
loss = (id_loss(text_recon_p, ids) + id_loss(visual_recon_p, ids)
+ id_loss(text_p, ids) + id_loss(visual_guided_p, ids))
# Ranking between reconstructed text and reconstructed image [§3.4] [eq. 7]
loss += ranking(text_recon_p, visual_recon_p, ids)
# Ranking between plain text and guided image [§3.4] [eq. 7]
loss += ranking(text_p, visual_guided_p, ids)
# Guidance: reconstructed features close to their guidance [§3.4]
loss += ranking(text_recon_p, text_p, ids)
loss += ranking(visual_recon_p, visual_guided_p, ids)
return loss # unit loss weights [§3.4]# Retrieval at test time: reconstructed features only [§3.4]
# The paper does not say how the K part vectors are combined; the code
# concatenates them.
def retrieval_scores(text_recon_p, visual_recon_p):
# Flatten the K part vectors of each sample into one vector
# [code: test_during_train.py#L16-L23]
text_vec = text_recon_p.flatten(start_dim=1) # (N_t, K*d')
visual_vec = visual_recon_p.flatten(start_dim=1) # (N_i, K*d')
# Cosine similarity matrix, rows are texts [§3.4]
return normalize(text_vec) @ normalize(visual_vec).THyperparameters
Values in the paper are the authors' own; "not stated" means the paper gives no value. Code values come from the official training script, which is what the authors ran (see Paper vs code).
| Name | Value | Where | Notes |
|---|---|---|---|
| Input image size | 384 x 128 | §4.2 dataloader.py | Code also pads 10 px and crops, see Paper vs code |
| Visual backbone | DeiT-Small, patch 16 | §4.2 model.py | Code loads the distilled ImageNet checkpoint |
| Feature width d | 384 | §4.2 DETR_model.py | Shared by visual, textual, dictionary and prototypes |
| Part width d' | 512 | §4.2 DETR_model.py | Per-prototype output width and classifier input |
| Dictionary atoms s | 400 | §4.2 DETR_model.py | Init not stated in paper; code uses N(0, 1) |
| Prototypes K | 6 | §4.2 train_mydecoder_pixelvit_txtimg_3_bert.py | Shared by image and text |
| Ranking margin alpha | 0.3 | §4.2 train_mydecoder_pixelvit_txtimg_3_bert.py | Cosine similarity |
| Bi-LSTM | 768 in, 384 per direction, no bias | text_feature_extract.py | Paper states only the Bi-LSTM and BERT input |
| Text encoder | BERT frozen; only Bi-LSTM trained | §4.2 text_feature_extract.py | Code uses bert-base-uncased |
| Mask | 1x1 map plus sigmoid, per token | §3.2 DETR_model.py | Code uses a linear layer, equivalent per token |
| Optimiser | Adam | §4.2 | |
| Adam betas | not stated | options.py | Code uses 0.9 and 0.999 |
| Weight decay | not stated | options.py | Defined as 4e-5 but not passed to Adam train_mydecoder_pixelvit_txtimg_3_bert.py |
| Batch size | 64 | §4.2 train_mydecoder_pixelvit_txtimg_3_bert.py | Training loader drops the last partial batch dataloader.py |
| Epochs | 60 | §4.2 train_mydecoder_pixelvit_txtimg_3_bert.py | |
| Learning rate, DeiT | 1e-4 | §4.2 train_mydecoder_pixelvit_txtimg_3_bert.py | Code uses 0.1 times the base rate 1e-3 |
| Learning rate, others | 1e-3 | §4.2 | Code gives decoders, dictionary, mask and prototypes 1e-4, see Paper vs code |
| LR schedule | not stated | train_mydecoder_pixelvit_txtimg_3_bert.py | Code: multi-step at epochs 20, 40, 50, default gamma 0.1 |
| Loss weights | not stated | train_mydecoder_pixelvit_txtimg_3_bert.py | Code: all unit weights |
| Decoder, dictionary (MHA_1) | depth 1, 6 heads, head dim 64, FFN 768, dropout 0 | DETR_model.py | Paper: one layer, no head count given |
| Decoder, prototypes (MHA_2) | depth 2, 6 heads, head dim 512, FFN 512, dropout 0 | DETR_model.py | Paper: one layer; MHA_2 "same structure as MHA_1" |
| Identity classes | 11,000 (CUHK); 3,102 (ICFG) | §4.1 train_mydecoder_pixelvit_txtimg_3_bert.py | Identification classifier per prototype |
| Random seed | not stated | train_mydecoder_pixelvit_txtimg_3_bert.py | Code seeds torch and Python random with 233; NumPy not seeded |
| Evaluation metric | Rank-1, Rank-5, Rank-10 | §4.1 | Rank-k: probability that some top-k candidate matches |
Results
Author-reported. Setup: CUHK-PEDES and ICFG-PEDES, 384 x 128 inputs, the backbone named in each row, with the paper's own baselines §4.1 §4.2. Rank-k accuracy in percent §4.1.
CUHK-PEDES, backbone ResNet-50 for the first block of rows, and DeiT-Small for LGUR tab. 1
| Method | Rank-1 | Rank-5 | Rank-10 |
|---|---|---|---|
| Dual Path [zheng2020dual] | 44.40 | 66.26 | 75.07 |
| CMPM/C [zhang2018deep] | 49.37 | - | 79.27 |
| MIA [niu2020improving] | 53.10 | 75.00 | 82.90 |
| A-GANet [liu2019deep] | 53.14 | 74.03 | 82.95 |
| GALM [jing2020pose] | 54.12 | 75.45 | 82.97 |
| TIMAM [sarafianos2019adversarial] | 54.51 | 77.56 | 84.78 |
| TDE [niu2020textual] | 55.25 | 77.46 | 84.56 |
| VTA [ge2019visual] | 55.32 | 77.00 | 84.26 |
| SCAN [lee2018stacked] | 55.86 | 75.97 | 83.69 |
| ViTAA [wang2020vitaa] | 55.97 | 75.84 | 83.52 |
| CMAAM [aggarwal2020text] | 56.68 | 77.18 | 84.86 |
| HGAN [zheng2020hierarchical] | 59.00 | 79.49 | 86.62 |
| NAFS [gao2021contextual] | 59.94 | 79.86 | 86.70 |
| DSSL [zhu2021dssl] | 59.98 | 80.41 | 87.56 |
| MGEL [wang2021text] | 60.27 | 80.01 | 86.74 |
| SSAN [ding2021semantically] | 61.37 | 80.15 | 86.73 |
| Han et al. [han2021text] | 61.65 | 80.98 | 86.78 |
| LapsCore [wu2021lapscore] | 63.40 | - | 87.80 |
| LGUR (ResNet-50) | 64.21 | 81.94 | 87.93 |
| LGUR (DeiT-Small) | 65.25 | 83.12 | 89.00 |
ICFG-PEDES tab. 2. Baseline numbers are cited from [ding2021semantically] (SSAN) rather than re-run §4.3.
| Method | Rank-1 | Rank-5 | Rank-10 |
|---|---|---|---|
| Dual Path | 38.99 | 59.44 | 68.41 |
| CMPM/C | 43.51 | 65.44 | 74.26 |
| MIA | 46.49 | 67.14 | 75.18 |
| SCAN | 50.05 | 69.65 | 77.21 |
| ViTAA | 50.98 | 68.79 | 75.78 |
| SSAN | 54.23 | 72.63 | 79.53 |
| LGUR (ResNet-50) | 57.42 | 74.97 | 81.45 |
| LGUR (DeiT-Small) | 59.02 | 75.32 | 81.56 |
Domain generalisation (train on one dataset, test on the other, no adaptation) tab. 3. The backbone is not stated for this table.
| Direction | Method | Rank-1 | Rank-5 | Rank-10 |
|---|---|---|---|---|
| C to I | Dual Path | 15.41 | 29.80 | 38.19 |
| C to I | MIA | 19.35 | 36.78 | 46.42 |
| C to I | SCAN | 21.27 | 39.26 | 48.83 |
| C to I | SSAN | 24.72 | 43.43 | 53.01 |
| C to I | SSAN (w/ BERT) | 29.24 | 49.00 | 58.53 |
| C to I | LGUR | 34.25 | 52.58 | 60.85 |
| I to C | Dual Path | 7.63 | 17.14 | 23.52 |
| I to C | MIA | 10.93 | 23.77 | 32.39 |
| I to C | SCAN | 13.63 | 28.61 | 37.05 |
| I to C | SSAN | 16.68 | 33.84 | 43.00 |
| I to C | SSAN (w/ BERT) | 21.07 | 38.94 | 48.54 |
| I to C | LGUR | 25.44 | 44.48 | 54.39 |
Cost (training time is per mini-batch, inference includes query features and similarity with the whole gallery; batch 64, 384 x 128, a Titan X GPU) tab. 4. Times in ms; the column CAM marks whether the method uses cross-modal attention.
| Dataset | CAM | Method | Train | Inference | Rank-1 |
|---|---|---|---|---|---|
| CUHK | yes | MIA | 680 | 42 | 53.10 |
| CUHK | yes | SCAN | 718 | 46 | 55.86 |
| CUHK | yes | NAFS | 1,284 | 42 | 59.94 |
| CUHK | no | Dual Path | 321 | 10 | 44.40 |
| CUHK | no | CMPM/C | 338 | 27 | 49.37 |
| CUHK | no | SSAN | 901 | 76 | 61.37 |
| CUHK | no | LGUR | 886 | 26 | 64.21 |
| ICFG | yes | MIA | 711 | 113 | 46.49 |
| ICFG | yes | SCAN | 738 | 114 | 50.05 |
| ICFG | yes | NAFS | 1,304 | 116 | - |
| ICFG | no | Dual Path | 342 | 11 | 38.99 |
| ICFG | no | CMPM/C | 356 | 31 | 43.51 |
| ICFG | no | SSAN | 973 | 77 | 54.23 |
| ICFG | no | LGUR | 910 | 31 | 57.42 |
Ablation on CUHK-PEDES and ICFG-PEDES, DeiT-Small backbone tab. 5. D = dictionary, M = foreground mask, G = guidance from text; "+ PGU" rows keep the PGU module; "+ DGA" rows keep D, M and G together.
| No. | Components | CUHK R-1 | CUHK R-5 | CUHK R-10 | ICFG R-1 | ICFG R-5 | ICFG R-10 |
|---|---|---|---|---|---|---|---|
| 0 | baseline | 58.67 | 79.08 | 85.82 | 52.09 | 70.91 | 78.07 |
| 1 | + PGU | 63.26 | 81.17 | 87.69 | 56.34 | 73.58 | 80.23 |
| 2 | + DGA (D, M, G) | 61.86 | 80.43 | 87.20 | 55.83 | 73.65 | 80.48 |
| 3 | + PGU + D | 64.28 | 81.95 | 88.11 | 57.52 | 74.84 | 81.14 |
| 4 | + PGU + D + M | 64.64 | 82.91 | 88.52 | 57.82 | 74.99 | 81.17 |
| 5 | + PGU + D + G | 64.80 | 82.29 | 88.32 | 58.17 | 75.83 | 81.51 |
| 6 | LGUR (all) | 65.25 | 83.12 | 89.00 | 59.02 | 75.32 | 81.56 |
Dictionary variants tab. 6. The row "w/ shared D (Ours)" matches configuration 3 of tab. 5, not the full model.
| Type of reconstruction | CUHK R-1 | CUHK R-5 | ICFG R-1 | ICFG R-5 |
|---|---|---|---|---|
| w/o reconstruction | 63.26 | 81.17 | 56.34 | 73.58 |
| w/ SA (self-attention) | 63.39 | 82.28 | 56.34 | 74.09 |
| w/ unshared D | 61.55 | 80.69 | 55.96 | 73.23 |
| w/ shared D (Ours) | 64.28 | 81.95 | 57.52 | 74.84 |
Prototype variants tab. 7. The row "w/ shared P (Ours)" equals the full model, configuration 6 of tab. 5.
| Type of P | CUHK R-1 | CUHK R-5 | ICFG R-1 | ICFG R-5 |
|---|---|---|---|---|
| w/ unshared P | 63.78 | 82.15 | 57.73 | 75.11 |
| w/ shared P (Ours) | 65.25 | 83.10 | 59.02 | 75.32 |
The text of §4.5 gives the same differences as the tables, for example PGU adding 4.59 points to the baseline on CUHK §4.5 tab. 5. The mask figure shows the mask highlights the pedestrian body and suppresses background fig. 3. The attention visualisation shows matched phrase-patch pairs with similar dictionary attention and unmatched pairs with different attention fig. 4.
Paper vs code
Official repository: https://github.com/ZhiyinShao-H/LGUR. The paper links it
[abstract]. The owner name matches the first author. The code was cloned at
commit c203cc7763abbc640d90b74aeb3986b73472410e. The repository has no LICENSE
file, so the code is not licensed for reuse; nothing is quoted from it, and only
path and line references are given.
Discrepancies between the paper and the code, each with both sides:
- Decoder depth. The paper says DGA and PGU each use one transformer layer §3.2 §3.3. The code builds the dictionary block with depth 1 [code: model/DETR_model.py#L144-L146] and the prototype block with depth 2 [code: model/DETR_model.py#L141-L143]. The paper says MHA_2 has the same structure as MHA_1 §3.3; the two blocks differ in head dimension (64 vs 512) and FFN width (768 vs 512) in the code.
- Residuals and normalisation. The paper defines MHA as multi-head attention followed by a feed-forward network, with no residuals or normalisation §3.2. The code uses pre-normalisation residual blocks [code: vit_pytorch/vit_pytorch.py#L147-L160] vit_pytorch.py, and one LayerNorm instance normalises both the query and the memory [code: vit_pytorch/vit_pytorch.py#L147-L153].
- Mask on guidance. Eq. 3 includes the mask M eq. 3. The ablation text says the guidance is used "without M" §4.5. The code applies the mask to the guidance path DETR_model.py. The code has no "without M" configuration.
- Ranking loss. The paper defines the ranking loss per matched pair, with one semi-hard negative per direction eq. 7 §3.4. The code computes a batch-level loss: for each anchor it draws one random semi-hard negative with NumPy, keeps anchors that have one, and sums over the batch in both directions [code: loss/RankingLoss.py#L31-L33] RankingLoss.py [code: loss/RankingLoss.py#L62-L79]. If no anchor has a semi-hard negative, one triplet is kept for the last index RankingLoss.py. The sum is not averaged, so its scale grows with batch size, while the identification losses are averaged.
- Loss wording. §3.2 says the textual reconstruction is "minimised" in similarity §3.2. The loss used in code is the ranking hinge, which pulls matched pairs together and pushes others apart RankingLoss.py. The paper's wording is ambiguous.
- Identification classifier. The paper does not describe the classifier eq. 6. The code uses one linear layer of width 512 to the identity count per prototype, initialised with standard deviation 0.001 Id_loss.py Id_loss.py. The image and text features share that layer [code: loss/Id_loss.py#L50-L53]. The loss sums the image and text cross-entropies Id_loss.py.
- Text input. The paper says the Bi-LSTM takes the input embeddings from BERT §3.1. The code feeds the last hidden states of bert-base-uncased [code: model/text_feature_extract.py#L68-L70] text_feature_extract.py.
- Augmentation. The paper uses only random horizontal flipping §4.2. The training transform also pads by 10 pixels and takes a random crop of 384 x 128 dataloader.py.
- Learning rates. The paper gives 1e-4 for DeiT and 1e-3 for "the others" §4.2. The code gives 1e-4 to the DeiT backbone [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L152], 1e-4 to the decoders, dictionary, mask and prototype queries [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L149], and 1e-3 to the Bi-LSTM, the per-prototype projections and the identification classifiers [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L153-L156].
- Schedule and weight decay. The paper states no schedule. The code applies a multi-step schedule at epochs 20, 40 and 50 with the default factor 0.1 [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L121] [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L177]. A weight-decay option exists options.py but is not passed to Adam [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L159], so the effective decay is zero.
- Backbones. The paper reports a ResNet-50 variant in the main tables tab. 1 tab. 2. The training entry point builds only the DeiT-Small backbone [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L138] DETR_model.py. No ResNet-50 configuration of this model is wired in the entry point, so those numbers cannot be checked against this code.
- Retrieval vector. The paper does not say how the K part vectors are combined for cosine similarity §3.4. The code concatenates them, giving a vector of 6 x 512 = 3072 values test_during_train.py [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L45-L49].
- Inference features. The paper says reconstructed features are used at inference §3.4. The code's test loop takes the second return value, the reconstructed features test_during_train.py [code: test_during_train.py#L252]. This agrees with the paper.
- Shared prototypes. The paper shares the prototypes between modalities §3.3. The code uses one query set for both, controlled by a flag [code: model/DETR_model.py#L187] train_mydecoder_pixelvit_txtimg_3_bert.py. This agrees with the paper.
- Unused parameters. The positional parameter and the patch-to-pixel rearrange are defined but never used in the forward pass DETR_model.py DETR_model.py. The text dropout is defined but not applied text_feature_extract.py.
- Hard-coded paths. The DeiT checkpoint path and the dataset roots point to a specific machine model.py [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L86-L96]. Running the code needs them edited.
- Determinism. Torch and Python random are seeded with 233 [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L16-L18]. NumPy is not seeded, and the semi-hard negatives are drawn with NumPy RankingLoss.py, so runs are not reproducible without more seeding.
Pitfalls
- Ambiguous part merging. The paper does not say how K part vectors form one retrieval vector; the code concatenates them test_during_train.py. Implementing a mean or a different order gives different results.
- Decoder configuration. The paper's "one layer" does not match the code's depths and head sizes (see Paper vs code). Implementing one layer everywhere will not reproduce the code's numbers.
- Missing sub-layer details. Residuals, layer norm, head count and FFN width are not in the paper. The code's choices are pre-norm residual blocks with six heads vit_pytorch.py.
- Loss scale. Unit weights are stated §3.4, but the code's ranking term is a batch sum over both directions, while identification losses are means [code: loss/RankingLoss.py#L76] train_mydecoder_pixelvit_txtimg_3_bert.py. This changes the balance between terms, and the paper does not say it.
- Mask has no supervision. The mask is learned only through the reconstruction and ranking losses; no mask loss appears in the training script [code: train_mydecoder_pixelvit_txtimg_3_bert.py#L220]. The paper says nothing about supervising it §3.2.
- Mask in ablation. The ablation row for guidance without the mask §4.5 has no matching configuration in the code (see Paper vs code). The 0.65-point gain from guidance on ICFG §4.5 therefore cannot be checked against this code.
- Ablation configurations. The "w/ shared D" row in tab. 6 is configuration 3 of tab. 5 (64.28), not the full model; the "w/ shared P" row in tab. 7 equals the full model (65.25) tab. 5 tab. 6 tab. 7. The paper does not say so.
- Semi-hard sampling. The code keeps only anchors with a semi-hard negative and draws one at random RankingLoss.py. If no anchor in a batch has one, a single triplet is kept RankingLoss.py. The paper does not describe either rule eq. 7.
- Numerical sensitivity. Cosine similarity is computed without an epsilon in training train_mydecoder_pixelvit_txtimg_3_bert.py but with one in the test code test_during_train.py. The two differ only for zero vectors.
- Shapes. The image has HW = 24 x 8 = 192 patch tokens for 384 x 128 with patch 16. This is derived, not stated in the paper. The class token is dropped [code: model/model.py#L174-L177].
- Costs hidden by the headline. The efficiency claim is for test time tab. 4. Training costs about 900 ms per batch of 64, similar to SSAN tab. 4. Timing is author-reported on one GPU model.
- Domain generalisation setup. The backbone for tab. 3 is not stated, and the paper gives no baselines beyond the listed methods §4.3.
- ICFG baselines. Baseline numbers on ICFG-PEDES are cited from SSAN rather than re-run §4.3.
Lineage
- Builds on: learned object queries in transformer decoders. The paper compares its prototypes to DETR object queries §3.3 [carion2020end]: DETR. Transformer attention [vaswani2017attention] and DeiT-Small [touvron2021training] have no card.
- Uses BERT: arXiv:1810.04805 (no card; linked on arXiv only).
- Compared against: NAFS, arXiv:2101.03036; SSAN, arXiv:2107.12666; and the methods in tab. 1 to tab. 3. Only cards in this wiki can feed concept pages; the others are linked on arXiv only.
- Superseded by: not stated in this wiki.
Source and license
Adapted from “Learning Granularity-Unified Representations for Text-to-Image Person Re-identification” by Zhiyin Shao, Xinyu Zhang, Meng Fang, Zhifeng Lin, Jian Wang, Changxing Ding (arXiv:2207.07802), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ZhiyinShao-H/LGUR at commit c203cc7763ab (none (no LICENSE file in the repository)). Any quoted code is shown under that license.