A Dual-Path Model With Adaptive Attention For Vehicle Re-Identification

In plain words

Re-identification means finding the same vehicle again in another camera's footage. This is hard because many cars look alike: same make, model and colour. What tells two such cars apart are small details, such as stickers, wheels, mirror lights or the badge on the front or back. Which details can be seen depends on the way the car faces the camera. A car seen from behind hides its front, so features taken from the front are misleading.

The model has two parts that run side by side. One looks at the whole car. The other first works out which way the car is pointing and where its main parts are, then looks only at the parts that are most likely to be visible. The two sets of features are merged into one vector per car, and two images are compared by how close their vectors are. A last step reorders the search results to improve them.

TL;DR

AAVER is a vehicle re-identification model with two parallel paths. The global path is a ResNet. The second path estimates 20 vehicle key-points and one of 8 orientations with a two-stage network, then selects the seven key-points of the most likely orientation group and pools features around them §3.2, §3.3. The paths are fused, trained with an L2-softmax loss and matched by cosine similarity, with optional re-ranking §3.1, §3.3, §3.4. Author-reported: 88.97% CMC@1 on VeRi-776 with ResNet-101 tab. 3. The official repository releases only the key-point and orientation model, not the re-identification network README.md.

Problem

Vehicles with different identities can share make, model and colour, so a global appearance feature can miss the subtle cues that separate them §1. Key-point attention that uses all key-points, as in the orientation-based approach of Wang et al. (2017), can be misled: a key-point hidden by the vehicle's orientation still contributes a non-zero response, which can pull attention to the wrong place §1 fig. 1. The authors state the aim as re-identification from still images without time or location cues §1. Training the key-point network needs key-point and orientation annotations, which the paper uses for VeRi-776 §5.1.

Method

AAVER has four parts §3:

1. Global features (§3.1). A ResNet-50 or ResNet-101 backbone, initialised from weights pre-trained on CompCars §3.1, produces a 2048-dimensional vector from its last convolutional layer. A shallow MLP maps this vector to the training feature. The identity loss is L2-softmax eq. 1: the feature is normalised and scaled by a trainable positive scalar α, so all features lie on a hypersphere of radius α §3.1.

2. Coarse key-point maps (§3.2, stage 1). A fully convolutional VGG-16 network predicts 21 maps at 56×56: 20 key-points and background §3.2. It is trained with a per-pixel cross-entropy over the 21 classes eq. 2. After this stage, its weights are frozen §3.2.

3. Refined key-point maps and orientation (§3.2, stage 2). A refinement network takes the 56×56 version of the image concatenated with the stage-1 maps. The paper describes it as a two-stack hourglass §3.2, with a parallel branch of two fully connected layers that predicts one of 8 orientation classes §3.2. It returns 20 refined foreground maps, with the background channel dropped §3.2. Training minimises L2 = L_H + λ·L_O eq. 3, where L_H is a squared-error heatmap loss eq. 4 and L_O a cross-entropy over the 8 orientations eq. 5. The paper sets λ = 10 by cross-validation §3.2.

4. Orientation-conditioned key-point selection (§3.3). There are eight orientation groups. Each group holds one orientation and its two neighbouring orientations §3.3 tab. 1. At inference the group with the highest likelihood is chosen, and its seven key-points are kept §3.3 tab. 1. For each kept key-point, a Gaussian with σ = 2 is placed at the peak of its map, to include the area around it §3.3. These maps (7 × 56 × 56) are concatenated with shallow features from the global ResNet, pooled after its Res2 block (256 × 56 × 56) §3.3 tab. 6. A second ResNet's Res3, Res4 and Res5 blocks turn this input into the localised feature f_l §3.3.

5. Fusion, loss and inference (§3.3, §3). The global feature f_g and the localised feature f_l are concatenated and passed through an MLP trained with L2-softmax §3.3 fig. 2. At test time the penultimate fully connected layer gives the vehicle representation §3. Vectors are compared by cosine similarity §3.3. Optionally, the initial ranking is refined with the re-ranking method of Zhong et al. (2017) §3.4.

Equations as written in the paper (symbols as defined there):

\mathcal{L}_S = -\log \frac{\exp(\mathbf{W}_y^{T}\,\alpha\mathbf{x}/\|\mathbf{x}\|_2 + b_y)}{\sum_{j=1}^{N}\exp(\mathbf{W}_j^{T}\,\alpha\mathbf{x}/\|\mathbf{x}\|_2 + b_j)} eq. 1

\mathcal{L}_1 = \frac{-1}{H W}\sum_{i=1}^{H}\sum_{j=1}^{W}\log\frac{\exp(\mathbf{x}_{i,j}(t^{*}_{i,j}))}{\sum_{k=1}^{N_1}\exp(\mathbf{x}_{i,j}(k))} eq. 2

\mathcal{L}_2 = \mathcal{L}_H + \lambda\,\mathcal{L}_O eq. 3

\mathcal{L}_H = \sum_{k=1}^{N_2}\sum_{i=1}^{H}\sum_{j=1}^{W}\left|h_k(i,j) - h^{*}_k(i,j)\right|^2 eq. 4

\mathcal{L}_O = -\log\frac{\exp(\mathbf{p}(p^{*}))}{\sum_{i=1}^{N_p}\exp(\mathbf{p}(i))} eq. 5

Pseudocode for inference, written from the paper's text. Shapes are for one image. The code release does not contain this part README.md.

# AAVER embedding of one vehicle image  [§3], [§3.3]
# Input:  image (224, 224, 3); small_image (56, 56, 3)
# Output: embedding vector; its size is not stated

def embed_vehicle(image, small_image):
    # 1. Coarse maps: 20 key-points plus background  [§3.2] [eq. 2]
    coarse = coarse_net(image)                       # (21, 56, 56)
    # 2. Refined key-point maps and orientation scores  [§3.2]
    fine, orientation = refine_net(
        concat(small_image, coarse))                 # (20,56,56), (8,)
    # 3. Score the 8 groups and keep the best one  [§3.3]
    #    The paper does not say how a group score is formed.
    group = best_orientation_group(orientation)
    # 4. The seven key-points of that group  [§3.3] [tab. 1]
    points = GROUP_KEYPOINTS[group]                  # 7 indices
    # 5. Gaussian (sigma = 2) at the peak of each selected map  [§3.3]
    selected = zeros(7, 56, 56)
    for k, p in enumerate(points):
        peak = argmax_2d(fine[p])                    # (row, col)
        selected[k] = gaussian_at(peak, sigma=2)
    # 6. Shallow features after Res2 of the global ResNet  [§3.3] [tab. 6]
    pooled = global_res2(image)                      # (256, 56, 56)
    # 7. Localised feature from Res3 to Res5 of a second ResNet  [§3.3]
    local = local_res3_to_res5(concat(selected, pooled))
    # 8. Global feature from the global ResNet  [§3.1]
    global_feat = global_resnet(image)               # (2048,)
    # 9. Fuse with a shallow MLP  [§3.3] [§3]
    return fusion_mlp(concat(global_feat, local))

Training, written from the paper's text and the released stage-1 and stage-2 losses:

# Stage 1: coarse key-point maps  [§3.2] [eq. 2]
# Input: image (224, 224, 3); label_map (56, 56) with classes 0 to 20

def train_stage1_step(image, label_map):
    # 1. Scores for 21 classes at each of the 56 x 56 pixels  [§3.2]
    scores = coarse_net(image)                       # (21, 56, 56)
    # 2. Per-pixel cross-entropy, averaged over pixels  [eq. 2]
    return pixel_cross_entropy(scores, label_map)

# Stage 2: refinement and orientation  [§3.2] [eq. 3-5]
# coarse_net is frozen after stage 1  [§3.2]

def train_stage2_step(image, small_image, target_maps, target_orient):
    # 1. Frozen coarse maps  [§3.2]
    coarse = coarse_net(image)                       # (21, 56, 56)
    # 2. Refined maps and orientation scores  [§3.2]
    fine, orient_scores = refine_net(
        concat(small_image, coarse))                 # (20,56,56), (8,)
    # 3. Heatmap regression loss  [eq. 4]
    heatmap_loss = squared_error(fine, target_maps)
    # 4. Orientation classification loss  [eq. 5]
    orient_loss = cross_entropy(orient_scores, target_orient)
    # 5. Weighted sum with lambda = 10, as stated  [eq. 3] [§3.2]
    return heatmap_loss + 10.0 * orient_loss

Hyperparameters

Values stated in the paper are cited to the paper. Values taken from the released code are marked as code and are not stated in the paper.

Results

Author-reported. Setup: VeRi-776 has 776 vehicles, with 576 identities for training (37,778 images) and 200 for testing (11,579 images), and 1,678 query images §4.1. VehicleID has 26,328 vehicles and 221,567 images. Comparisons use the 800, 1600 and 2400 gallery sizes §4.1. Hardware and training time are not stated.

Key-point and orientation network, against the orientation-based method of Wang et al. (2017) tab. 2:

VeRi-776, global baseline against AAVER with the same backbone tab. 3:

VehicleID, CMC scores on the three gallery sizes tab. 4:

Comparison with other methods, ResNet-101 for AAVER tab. 5. "-" means not reported:

Ablations on VeRi-776 and VehicleID (VehicleID at gallery 800 only). Depth of the pooled global layer tab. 6; mAP is a fraction here:

Single branch against dual branch tab. 7:

Heatmap use: all key-point maps combined into one, as in Wang et al., against the adaptive selection tab. 8:

Supplement, VeRi-Wild, which is a larger dataset from 174 cameras tab. 9. The table uses the same protocol as VeRi-776; splits are Small, Medium and Large:

Headline reading: the abstract's "more than 7%" gain for key-point prediction matches the relative gain at r0 = 3 (95.30 / 88.8, about 7.3%). The absolute difference is 6.5 points tab. 2. Re-ranking raises VeRi-776 mAP from 61.18 to 66.35 tab. 5. The paper says AAVER is state of the art on most metrics, but RAM is higher on VehicleID CMC@1 at all three gallery sizes tab. 5.

Paper vs code

The official repository is https://github.com/Pirazh/Vehicle_Key_Point_Orientation_Estimation, owned by the first author. It is linked from the abstract and it holds the key-point and orientation model only. It has no LICENSE file. Its README states that the re-identification code is not in the repository README.md. So the code cannot be quoted, and only the pseudocode above is reused from it.

What was searched: the arXiv abstract and the paper's text (the single code link); the author's GitHub repository list (11 repositories). No repository for AAVER or for the re-identification model was found. The author's SSBVER repository is a different paper's method and was not used.

Differences between the paper and the released code, each with both citations:

• Scope. The paper's full pipeline includes the global ResNet, CompCars pre-training, the localised branch, the adaptive selection, the L2-softmax identity loss and re-ranking §3. None of these are in the release. Only the two-stage key-point and orientation network is there README.md KP_Orientation_Net.py.
• λ. The paper uses λ = 10 §3.2. The code default is 0.1 main.py, and the README commands do not change it README.md README.md.
• Heatmap loss reduction. The paper's eq. 4 sums over key-points and pixels eq. 4. The code uses nn.MSELoss(), which averages over batch, key-points and pixels train.py train.py. The averaging divides by the batch size and by 20 × 56 × 56 = 62,720 per image, so λ = 10 means something different in the code.
• Stage-1 weighting. The paper's eq. 2 averages unweighted over pixels eq. 2. The code weights each class by its inverse frequency train.py veri_dataset.py.
• Orientation weighting. The code also weights the orientation cross-entropy by inverse class frequency train.py. The paper does not mention this.
• Normalisation. The paper uses ImageNet mean and std §4.2. The code uses the mean and std of VeRi-776 paths.py transforms.py.
• Batch size and epochs. The paper uses batch 150 and 40, 40, 20 and 20 epochs §4.2. The code default is batch 128 main.py and 10 epochs main.py. The README trains each stage for 15 epochs README.md README.md.
• Refinement network. The paper describes a two-stack hourglass §3.2. The forward pass is a single pass through the layers, with one set of residual additions. No second stack is visible KP_Orientation_Net.py.
• Backbone. The paper names VGG-16 §3.2. The code uses vgg16_bn, which adds BatchNorm, and it adds skip connections from the 14×14 and 28×28 stages that the paper does not describe KP_Orientation_Net.py KP_Orientation_Net.py.
• Dropout. The orientation head uses dropout of 0.5, which the paper does not state KP_Orientation_Net.py.
• Error metric. The paper reports "MSE in pixels" §5.1. The code computes the mean Euclidean distance between predicted and true peaks, not a squared error utilities.py. It averages per batch and then over batches test.py test.py.
• Precision protocol. The second half of Table 2 uses a distance threshold r0 on the 48 × 48 grid, following Wang et al. §5.1 tab. 2. This threshold is not in the release. The release reports only the distance metric above test.py.
• Target maps. The code's targets are one-hot points, one pixel per key-point, and the Gaussian dilation is commented out veri_dataset.py. The paper does not describe the target maps.
• Fine-map normalisation. The refinement output is a softmax over spatial positions for each channel KP_Orientation_Net.py. The paper does not say this.
• Coarse input to stage 2. The stage-2 input is the raw coarse output, which is non-negative after ReLU and not normalised KP_Orientation_Net.py KP_Orientation_Net.py.
• Key-point labels. The code indexes key-points from 0 with the names in KP_labels utilities.py. The numbers in the paper's Table 1 look 1-based: under that reading, the Rear group contains rear lamps, logos, plate and mirrors tab. 1. The paper does not state its indexing, so this is an inference.

Pitfalls

• Group score is not defined. The paper says the group with the highest likelihood is chosen §3.3. It does not say how a group's likelihood is computed from the 8 class probabilities, for example as a sum over its three classes. The release has no selection code.
• Gaussian step is ambiguous. The paper says a Gaussian with σ = 2 is placed at each peak §3.3. It does not say whether the result is a bare Gaussian or the heatmap convolved with one, and σ is in unstated units.
• "Pooled" features are not defined. The paper says the shallow features are pooled §3.3, but the stated size is the full 56 × 56 map tab. 6. Read it as the full Res2 map unless the authors say otherwise.
• Dimensions are missing. The MLP widths, the embedding size and the initial α of the L2-softmax layer are not stated §3.1 §3.3.
• Re-ranking. The parameters of the Zhong et al. method are not stated §3.4. Gains reported with it cannot be reproduced from this card.
• Target maps and loss scale. One-hot targets with the squared error of eq. 4 give a different loss scale from Gaussian targets. The code uses one-hot targets, and its loss is a mean, not a sum veri_dataset.py train.py. Keep the reduction and the target form together when reproducing λ.
• Stage-2 inputs must match. The coarse maps are raw outputs, but the refined maps are softmax-normalised over space KP_Orientation_Net.py KP_Orientation_Net.py. Mixing them up changes the input distribution that stage 2 sees.
• Flip augmentation is incomplete. The code flips x-coordinates and remaps orientation classes, but it does not swap left and right key-point indices veri_dataset.py. Left and right parts would be mislabelled if flipping is enabled. Flipping is off by default main.py.
• Model selection uses the test set. The released training evaluates the test split after each epoch and keeps the best checkpoint by test error train.py train.py. The paper does not say this. Reported test numbers may be optimistic.
• Orientation boundaries are soft. The paper says there is no absolute boundary between adjacent orientations, and gives right and right front as an example of confusion §3.3 fig. 4. The 84.44% accuracy tab. 2 includes these errors.
• Dataset access. VeRi-776 images are not in the repository. The key-point annotations are in data/VehicleKeyPointData, and the README points to an external annotation source README.md. The licence of that data was not checked here.
• Compute is not stated. The paper gives no runtime or parameter counts. The method runs a VGG-16, a refinement network, and two ResNets per image.
• Table units differ. Tables 3 to 5 report percentages. Tables 6 to 8 report mAP as a fraction tab. 6 tab. 7 tab. 8.

Lineage

• Builds on: the orientation-based key-point attention of Wang et al. (2017), which the paper extends with orientation-conditioned selection §1 §2. The two-stage key-point design follows facial key-point detection (Kumar and Chellappa, 2018) and human pose estimation (Bulat et al., 2016) §1. The refinement follows the stacked hourglass network (Newell et al., 2016) §3.2. The identity loss is L2-softmax (Ranjan et al., 2017) §3.1. Re-ranking is from Zhong et al. (2017) §3.4. Shallow feature content follows Zeiler and Fergus (2013) §3.3.
• Compared against: SCPL, OIFE, VAMI and RAM tab. 5, and the single-branch and combined-heatmap variants described in the ablations tab. 7 tab. 8.
• Superseded by: not stated in this wiki.