Siamese network

Overview

A Siamese network applies one encoder, with tied weights, to two inputs and then compares what comes out 1908.10084 §3 2201.01293 §2. The shared weights mean both inputs are read the same way. The three papers in this wiki use it for different ends. Sentence-BERT encodes each sentence on its own into a vector, so a comparison is a cheap calculation on stored vectors 1908.10084 §3. ChangeFormer keeps feature maps of both images, compares them at several scales, and decodes a map of changed pixels 2201.01293 §2.1.3 2201.01293 §2.2. SimTrack is the contrast case: it drops the separate branches and lets one backbone attend across the exemplar and search inputs 2203.05328 §3.2.

Variants

Sentence embeddings compared by cosine similarity

Two copies of BERT or RoBERTa share weights, and mean pooling over the real tokens gives one vector per sentence 1908.10084 §3. Training uses one of three objectives: a softmax over the pair's combined features for NLI labels, a mean squared error between cosine similarity and a graded score, or a triplet hinge on Euclidean distance 1908.10084 §3. The NLI head is dropped at inference, and similarity is the cosine of the pooled vectors 1908.10084 §6. The authors report an average Spearman of 76.55 over seven STS tasks for the NLI-trained large model, without STS training data 1908.10084 tab. 1. Card: Sentence-BERT.

Change masks from a Siamese transformer

One hierarchical transformer encoder reads the pre-change and post-change images 2201.01293 §2. At each of four scales, a difference module concatenates the two feature maps and applies a 3×3 convolution, ReLU and batch normalisation, so the comparison is learned rather than a fixed absolute difference 2201.01293 §2.1.3. A light MLP decoder turns the four difference maps into a two-class change mask 2201.01293 §2.2. The attention keys and values are shortened by a reduction ratio R, which lowers the cost of attention 2201.01293 §2.1.1. The authors report F1 90.40 on LEVIR-CD and 86.67 on DSIFN-CD, above BIT on both 2201.01293 tab. 1. Card: A Transformer-Based Siamese Network for Change Detection.

Joint encoding in one ViT for tracking

SimTrack starts from a Siamese tracker, in which the marked first frame and the search frame go through separate branches and a transformer head compares them 2203.05328 §1. It concatenates the exemplar tokens and search tokens along the sequence and runs them through one ViT backbone 2203.05328 §3.2, eq. 4. Every layer's attention includes both exemplar-to-search and search-to-exemplar blocks 2203.05328 eq. 5, eq. 6, so the comparison happens inside the backbone. The transformer head is removed, and the search outputs go straight to a corner predictor 2203.05328 §3.2. Exemplar tokens get their own learned position embedding, computed from grid position and the target-area fraction in each patch 2203.05328 §3.2, eq. 7. A 64×64 centre crop of the exemplar adds extra input-only tokens for finer detail 2203.05328 §3.3. The authors report 69.3% AUC on LaSOT and 54.8% on TNL2K for ViT-B/16 [arxiv:2203.05328 tab. 1]. Card: Backbone is All Your Need.

When to use which

Implementation sketch

The Siamese core is one encoder run twice, then a comparison. The variants differ in the comparison and in what is trained. The joint tracking variant instead runs one pass over the concatenated inputs.

# Siamese comparison: one encoder, two inputs  [arxiv:1908.10084 §3] [arxiv:2201.01293 §2]
# Input: input_a, input_b; output: a score or a mask
def siamese(input_a, input_b, encoder, compare):
    # The same weights read both inputs  [arxiv:1908.10084 §3]
    feat_a = encoder(input_a)
    feat_b = encoder(input_b)
    # Variant choice: cosine of pooled vectors, or learned difference maps
    return compare(feat_a, feat_b)

# Sentence variant: embed once, compare later  [arxiv:1908.10084 §3] [§6]
def sentence_embedding(token_ids, mask):
    # Average the token outputs over real tokens, not padding  [arxiv:1908.10084 §3]
    tokens = encoder(token_ids, mask)                 # (B, T, n)
    return masked_mean(tokens, mask)                  # (B, n)

def cosine(u, v):
    # Cosine of two stored vectors; the NLI head is not used here  [arxiv:1908.10084 §6]
    return dot(u, v) / (norm(u) * norm(v))            # scalar

# Change variant: learned difference at each scale  [arxiv:2201.01293 §2.1.3]
def difference(feat_pre, feat_post):
    # Combine both maps of one scale, then conv, ReLU and batch norm  [arxiv:2201.01293 §2.1.3]
    fused = concat(feat_pre, feat_post, axis=channels)
    return batch_norm(relu(conv3x3(fused)))           # (B, C, H, W)

# Joint tracking variant: one backbone over both inputs  [arxiv:2203.05328 §3.2]
def joint_forward(exemplar_tokens, search_tokens, backbone):
    # Concatenate along the sequence, so attention mixes both  [arxiv:2203.05328 eq. 4]
    x = concat(exemplar_tokens, search_tokens, axis=sequence)
    x = backbone(x)
    # Only the search outputs go to the corner predictor  [arxiv:2203.05328 §3.2]
    return x[len(exemplar_tokens):]

When you train the sentence variant with triplets, set the margin explicitly. The paper uses 1 1908.10084 §3, but the released library's default is a different value, see the SBERT card.

Contradictions

The three papers address different tasks, so we know of no direct disagreement between them on the Siamese design itself.

Within the ChangeFormer card, the text in §4 quotes DSIFN-CD gains over BIT of about 20.0 percent for F1 and 6.4 percent for OA. Table 1 gives 25.1 and 6.9 percent relative gains for those two metrics 2201.01293 §4 2201.01293 tab. 1. The LEVIR-CD gains agree with the table. Use the table values, and note the mismatch when you cite them.

For SimTrack, the paper gives eq. 7 as the exemplar position input, but the released code computes that input differently, from box overlap rather than target area 2203.05328 §3.2, eq. 7. See the SimTrack card for the code lines. A literal implementation of the paper will not reproduce the released model's inputs.

Papers

• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks: fine-tunes BERT and RoBERTa with shared weights and mean pooling, so sentences can be compared by cosine similarity.
• A Transformer-Based Siamese Network for Change Detection: a Siamese hierarchical transformer with per-scale learned difference modules and an MLP decoder for change masks.
• Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking: one ViT over concatenated exemplar and search tokens replaces the Siamese branches and transformer head.