---
kind: concept
slug: siamese-network
title: Siamese network
summary: >-
  One encoder with shared weights reads two inputs, then they are compared. Variants: cosine similarity between stored sentence vectors, learned per-scale differences for change masks, or one joint ViT over both inputs that drops the separate branches. Pick by whether you need a reusable vector or a dense output, and by labels and compute.
tags: [siamese-network, metric-learning, sentence-embeddings, change-detection, visual-object-tracking, transformer]
papers: ["1908.10084", "2201.01293", "2203.05328"]
related: [text-embedding, metric-learning, contrastive-learning, vision-transformer, multi-scale-feature-fusion, self-attention, nearest-neighbor-search, visual-object-tracking, positional-encoding]
status: draft
updated_at: 2026-11-01
---

## Overview

A Siamese network applies one encoder, with tied weights, to two inputs and
then compares what comes out [arxiv:1908.10084 §3] [arxiv:2201.01293 §2]. The
shared weights mean both inputs are read the same way. The three papers in this
wiki use it for different ends. Sentence-BERT encodes each sentence on its own
into a vector, so a comparison is a cheap calculation on stored vectors
[arxiv:1908.10084 §3]. ChangeFormer keeps feature maps of both images, compares
them at several scales, and decodes a map of changed pixels
[arxiv:2201.01293 §2.1.3] [arxiv:2201.01293 §2.2]. SimTrack is the contrast
case: it drops the separate branches and lets one backbone attend across the
exemplar and search inputs [arxiv:2203.05328 §3.2].

## Variants

### Sentence embeddings compared by cosine similarity

Two copies of BERT or RoBERTa share weights, and mean pooling over the real
tokens gives one vector per sentence [arxiv:1908.10084 §3]. Training uses one
of three objectives: a softmax over the pair's combined features for NLI
labels, a mean squared error between cosine similarity and a graded score, or
a triplet hinge on Euclidean distance [arxiv:1908.10084 §3]. The NLI head is
dropped at inference, and similarity is the cosine of the pooled vectors
[arxiv:1908.10084 §6]. The authors report an average Spearman of 76.55 over
seven STS tasks for the NLI-trained large model, without STS training data
[arxiv:1908.10084 tab. 1]. Card: [Sentence-BERT](/papers/1908.10084).

### Change masks from a Siamese transformer

One hierarchical transformer encoder reads the pre-change and post-change
images [arxiv:2201.01293 §2]. At each of four scales, a difference module
concatenates the two feature maps and applies a 3×3 convolution, ReLU and
batch normalisation, so the comparison is learned rather than a fixed
absolute difference [arxiv:2201.01293 §2.1.3]. A light MLP decoder turns the
four difference maps into a two-class change mask [arxiv:2201.01293 §2.2]. The
attention keys and values are shortened by a reduction ratio R, which lowers
the cost of attention [arxiv:2201.01293 §2.1.1]. The authors report F1 90.40
on LEVIR-CD and 86.67 on DSIFN-CD, above BIT on both [arxiv:2201.01293 tab. 1].
Card: [A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293).

### Joint encoding in one ViT for tracking

SimTrack starts from a Siamese tracker, in which the marked first frame and
the search frame go through separate branches and a transformer head compares
them [arxiv:2203.05328 §1]. It concatenates the exemplar tokens and search
tokens along the sequence and runs them through one ViT backbone
[arxiv:2203.05328 §3.2, eq. 4]. Every layer's attention includes both
exemplar-to-search and search-to-exemplar blocks [arxiv:2203.05328 eq. 5, eq. 6],
so the comparison happens inside the backbone. The transformer head is
removed, and the search outputs go straight to a corner predictor
[arxiv:2203.05328 §3.2]. Exemplar tokens get their own learned position
embedding, computed from grid position and the target-area fraction in each
patch [arxiv:2203.05328 §3.2, eq. 7]. A 64×64 centre crop of the exemplar adds
extra input-only tokens for finer detail [arxiv:2203.05328 §3.3]. The authors
report 69.3% AUC on LaSOT and 54.8% on TNL2K for ViT-B/16 [arxiv:2203.05328
tab. 1]. Card: [Backbone is All Your Need](/papers/2203.05328).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Many texts to search or cluster | Sentence embeddings with cosine similarity | Each text is encoded once. For 10,000 sentences the authors estimate about 5 s instead of about 65 h with a BERT pair model [arxiv:1908.10084 §1]. |
| A few known pairs to score, accuracy first | A pair model (BERT reading both inputs) | On STS-benchmark-trained rows, BERT scores above SBERT, e.g. 88.77 vs 86.10 for large NLI+STSb [arxiv:1908.10084 tab. 2]. Each pair costs one full pass. |
| Labelled pairs for NLI-style or graded similarity | Sentence embeddings, NLI-trained | NLI data alone gives 76.55 average over seven STS tasks [arxiv:1908.10084 tab. 1]. |
| No labelled data at all | Not covered by these papers | The paper does not report a result without labels, so treat this as untested. |
| Raw BERT outputs as sentence vectors | Avoid | Mean-pooled BERT averages 54.81 on the seven STS tasks, below averaged GloVe at 61.32 [arxiv:1908.10084 tab. 1]. |
| Pixel-level labelled change masks for image pairs | Siamese transformer with difference modules | It beats BIT on the two reported datasets [arxiv:2201.01293 tab. 1]. Both datasets are remote-sensing sets with patch splits [arxiv:2201.01293 §3.1]. |
| Large images and tight GPU memory | Siamese transformer with a reduction ratio | Attention cost drops from quadratic in pixels to quadratic divided by R [arxiv:2201.01293 §2.1.1]. The paper does not give R, a parameter count or FLOPs, so measure them yourself. |
| Reproducing the reported change-mask numbers | Start from the released settings, not the paper alone | The paper says the network is randomly initialised [arxiv:2201.01293 §3.2]. Released scripts load pretrained weights, so the numbers may depend on that choice. |
| Single-object tracking with one pre-trained ViT and fewer custom modules | Joint ViT over exemplar and search tokens | The paper removes the correlation and transformer head [arxiv:2203.05328 §3.2]. ViT-B/16 at 25.0 GFLOPs gets 69.3 LaSOT AUC, against 64.9 at 29.3 GFLOPs for TransT [arxiv:2203.05328 tab. 1]. The rows use different backbones and input sizes, so this is not a like-for-like test. |
| Tracking when the exemplar must be encoded once and reused | Siamese branches | The joint design runs the exemplar through the backbone together with each search frame [arxiv:2203.05328 eq. 4]. The paper reports no per-frame latency, so measure it before you choose. |

## Implementation sketch

The Siamese core is one encoder run twice, then a comparison. The variants
differ in the comparison and in what is trained. The joint tracking variant
instead runs one pass over the concatenated inputs.

```pseudocode
# Siamese comparison: one encoder, two inputs  [arxiv:1908.10084 §3] [arxiv:2201.01293 §2]
# Input: input_a, input_b; output: a score or a mask
def siamese(input_a, input_b, encoder, compare):
    # The same weights read both inputs  [arxiv:1908.10084 §3]
    feat_a = encoder(input_a)
    feat_b = encoder(input_b)
    # Variant choice: cosine of pooled vectors, or learned difference maps
    return compare(feat_a, feat_b)
```

```pseudocode
# Sentence variant: embed once, compare later  [arxiv:1908.10084 §3] [§6]
def sentence_embedding(token_ids, mask):
    # Average the token outputs over real tokens, not padding  [arxiv:1908.10084 §3]
    tokens = encoder(token_ids, mask)                 # (B, T, n)
    return masked_mean(tokens, mask)                  # (B, n)

def cosine(u, v):
    # Cosine of two stored vectors; the NLI head is not used here  [arxiv:1908.10084 §6]
    return dot(u, v) / (norm(u) * norm(v))            # scalar
```

```pseudocode
# Change variant: learned difference at each scale  [arxiv:2201.01293 §2.1.3]
def difference(feat_pre, feat_post):
    # Combine both maps of one scale, then conv, ReLU and batch norm  [arxiv:2201.01293 §2.1.3]
    fused = concat(feat_pre, feat_post, axis=channels)
    return batch_norm(relu(conv3x3(fused)))           # (B, C, H, W)
```

```pseudocode
# Joint tracking variant: one backbone over both inputs  [arxiv:2203.05328 §3.2]
def joint_forward(exemplar_tokens, search_tokens, backbone):
    # Concatenate along the sequence, so attention mixes both  [arxiv:2203.05328 eq. 4]
    x = concat(exemplar_tokens, search_tokens, axis=sequence)
    x = backbone(x)
    # Only the search outputs go to the corner predictor  [arxiv:2203.05328 §3.2]
    return x[len(exemplar_tokens):]
```

When you train the sentence variant with triplets, set the margin
explicitly. The paper uses 1 [arxiv:1908.10084 §3], but the released library's
default is a different value, see the [SBERT card](/papers/1908.10084).

## Contradictions

The three papers address different tasks, so we know of no direct
disagreement between them on the Siamese design itself.

Within the ChangeFormer card, the text in §4 quotes DSIFN-CD gains over BIT of
about 20.0 percent for F1 and 6.4 percent for OA. Table 1 gives 25.1 and 6.9
percent relative gains for those two metrics [arxiv:2201.01293 §4] [arxiv:2201.01293 tab. 1].
The LEVIR-CD gains agree with the table. Use the table values, and note the
mismatch when you cite them.

For SimTrack, the paper gives eq. 7 as the exemplar position input, but the
released code computes that input differently, from box overlap rather than
target area [arxiv:2203.05328 §3.2, eq. 7]. See the
[SimTrack card](/papers/2203.05328) for the code lines. A literal
implementation of the paper will not reproduce the released model's inputs.

## Papers

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](/papers/1908.10084): fine-tunes BERT and RoBERTa with shared weights and mean pooling, so sentences can be compared by cosine similarity.
- [A Transformer-Based Siamese Network for Change Detection](/papers/2201.01293): a Siamese hierarchical transformer with per-scale learned difference modules and an MLP decoder for change masks.
- [Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking](/papers/2203.05328): one ViT over concatenated exemplar and search tokens replaces the Siamese branches and transformer head.