A Transformer-Based Siamese Network for Change Detection

In plain words

Change detection takes two photos of the same place, taken at different times, and marks the pixels where something has changed, such as a new building or cleared land. Shadows, seasons and lighting shifts count as noise and should be ignored.

Most systems for this use convolutional networks, which look at small neighbourhoods and build up context slowly. ChangeFormer uses a transformer instead, which can relate distant parts of an image directly. The same encoder reads both photos, the features it finds are compared at several scales, and a small decoder turns the comparison into a map of changed and unchanged pixels. The authors argue that a heavy convolutional backbone is not needed for this task. Their reported results beat the earlier methods they compare with on two public datasets.

TL;DR

• ChangeFormer is a Siamese transformer for change detection between two co-registered images. One hierarchical encoder with shared weights reads both images, four per-scale difference modules compare them, and a lightweight MLP decoder outputs a two-class change mask §2 fig. 1.
• Attention keys and values are shortened by a reduction ratio R, which lowers the cost from O((HW)²) to O((HW)²/R). The paper does not state R numerically §2.1.1 eq. 2.
• Author-reported: best F1, IoU and OA of the nine rows in the table on both LEVIR-CD and DSIFN-CD, for example F1 90.40 and 86.67 tab. 1.
• Do not implement from the paper alone. The released code differs in the decoder, the downsampling kernels, the difference module, the loss, the augmentation and the initialisation. See Paper vs code.

Problem

• The input is a pair of co-registered remote-sensing images taken at different times. The goal is to find relevant changes, such as buildings, vehicles and vegetation, and to ignore irrelevant ones, such as seasonal variation, building shadows and illumination changes §1.
• Convolutional change detectors widen their receptive field with stacked, dilated or attention-based layers. The authors say attention re-weights ConvNet features along channel and spatial axes and so struggles to relate long-range details across space and time §1.
• Transformers have a larger effective receptive field than deep ConvNets §1. The earlier transformer method, BIT, still runs a ResNet18 encoder under a transformer §1 tab. 1. The paper's claim is that a hierarchical transformer encoder with a lightweight MLP decoder can replace the ConvNet backbone §1 §5.
• Full self-attention costs O((HW)²) in time and memory, which rules it out for high-resolution inputs §2.1.1 eq. 1.

Method

The network has three modules: a hierarchical transformer encoder in a Siamese arrangement, four difference modules, and an MLP decoder §2 fig. 1.

Encoder

• The encoder has four stages. Stage i outputs a map at resolution H/2^(i+1), so the four maps are at 1/4, 1/8, 1/16 and 1/32 of the input, with channel counts that increase with i §2.1.
• Each stage starts with an overlapping convolutional patch embedding, then runs transformer blocks. Stage 1 uses kernel K=7, stride S=4, padding P=3. The paper says the other stages use K=3, S=2, P=1 §2.1.2. The released code differs here; see Paper vs code.
• A transformer block applies self-attention with sequence reduction eq. 1 eq. 2 eq. 3, then a positional MLP that contains a 3×3 convolution eq. 4. Position is supplied by that convolution rather than by fixed embeddings, so the network can take test images of a different size from the training images §2.1.1.

# Siamese change detector; both images use the same encoder  [§2] [fig. 1]
# Input: pre, post as (B, 3, H, W); output: logits (B, 2, H, W)
def changeformer(pre, post):
    # Encode each image into four maps at 1/4, 1/8, 1/16, 1/32  [§2.1]
    maps_pre = encode(pre)
    maps_post = encode(post)
    # One learned difference per scale  [§2.1.3] [eq. 5]
    diffs = [
        difference_module(feat_pre, feat_post)
        for feat_pre, feat_post in zip(maps_pre, maps_post)
    ]
    return mlp_decoder(diffs)  # [§2.2]

# Hierarchical encoder: four stages of downsampling then transformer blocks
# [§2.1] [§2.1.2]
def encode(image):
    maps = []
    for stage in range(4):  # i = 1..4, channel count grows with i  [§2.1]
        # Stage 1: K=7, S=4, P=3. Stages 2-4: K=3, S=2, P=1  [§2.1.2]
        tokens, height, width = overlap_patch_embed[stage](image)
        for block in transformer_blocks[stage]:
            tokens = transformer_block(tokens, height, width)  # [§2.1.1]
        image = to_map(tokens, height, width)  # (B, C_i, H, W)
        maps.append(image)
    return maps

# Transformer block: attention, then positional MLP  [§2.1.1]
# The paper writes no residual or normalisation here; see Pitfalls.
def transformer_block(tokens, height, width):
    attended = reduced_attention(tokens)  # [eq. 1] [eq. 2] [eq. 3]
    return positional_mlp(attended, height, width)  # [eq. 4]

# Self-attention with a shorter key and value sequence  [eq. 1] [§2.1.1]
# Queries keep length HW; the paper also reduces Q, see Pitfalls.
def reduced_attention(tokens):
    queries = tokens  # (B, HW, C)
    keys = reduce_sequence(tokens)  # (B, HW/R, C)
    values = reduce_sequence(tokens)  # (B, HW/R, C)
    # Scaled dot-product attention  [eq. 1]
    scores = queries @ transpose(keys) / sqrt(head_dim)  # (B, HW, HW/R)
    return softmax(scores) @ values  # (B, HW, C)

# Sequence reduction: group R tokens and project back to C  [eq. 2] [eq. 3]
# R is not stated numerically in the paper  [§2.1.1]
def reduce_sequence(tokens):
    grouped = reshape(tokens, (HW // R, C * R))  # [eq. 2]
    return linear(C * R, C)(grouped)  # [eq. 3], (B, HW/R, C)

# Positional MLP: MLP, 3x3 conv, GELU, MLP, then a residual  [eq. 4]
# The convolution provides position information  [§2.1.1]
def positional_mlp(features, height, width):
    hidden = linear_in(features)  # (B, HW, C)
    hidden = conv3x3(to_map(hidden, height, width))  # [eq. 4]
    hidden = gelu(to_tokens(hidden))  # back to (B, HW, C)
    return linear_out(hidden) + features  # residual  [eq. 4]

The equations, as written in the paper, are:

\text{Attention}(Q,K,V) = \text{Softmax}\left(\frac{QK^{T}}{\sqrt{d_{\text{head}}}}\right)V eq. 1

\hat{S} = \text{Reshape}\left(\frac{HW}{R},\, C \cdot R\right) S eq. 2

S = \text{Linear}(C \cdot R,\, C)\,\hat{S} eq. 3

F_{\text{out}} = \text{MLP}\big(\text{GELU}(\text{Conv2D}_{3\times3}(\text{MLP}(F_{\text{in}})))\big) + F_{\text{in}} eq. 4

Difference modules

• There is one difference module per stage. It concatenates the pre-change and post-change maps of that stage and applies a 3×3 convolution, a ReLU and batch normalisation eq. 5 §2.1.3.
• The module learns its distance measure rather than taking an absolute difference, which the authors say improves CD performance over the absolute difference used by BIT §2.1.3.

F^{i}_{\text{diff}} = \text{BN}\big(\text{ReLU}(\text{Conv2D}_{3\times3}(\text{Cat}(F^{i}_{\text{pre}}, F^{i}_{\text{post}})))\big) eq. 5

# Difference module for one scale i  [§2.1.3] [eq. 5]
# Learns the comparison from the concatenated pair; no absolute difference
def difference_module(feat_pre, feat_post):
    fused = concatenate(feat_pre, feat_post, axis=channels)  # (B, 2C_i, H, W)
    return batch_norm(relu(conv3x3(fused)))  # output channels not stated

MLP decoder

1. Each difference map goes through a per-pixel linear layer to the embedding size C_ebd, then bilinear resizing to H/4 × W/4 eq. 6 eq. 7 §2.2.1.
2. The four resized maps are concatenated and fused by a linear layer back to C_ebd eq. 8 §2.2.2.
3. A transposed convolution with stride 4 and kernel 3 brings the map to H × W, and a per-pixel linear layer outputs the two logits, change and no-change eq. 9 eq. 10 §2.2.3.

\tilde{F}^{i}_{\text{diff}} = \text{Linear}(C_i, C_{\text{ebd}})(F^{i}_{\text{diff}}) \quad \forall i eq. 6

\hat{F}^{i}_{\text{diff}} = \text{Upsample}\big((H/4, W/4), \text{bilinear}\big)(\tilde{F}^{i}_{\text{diff}}) eq. 7

F = \text{Linear}(4C_{\text{ebd}}, C_{\text{ebd}})\big(\text{Cat}(\hat{F}^{1}_{\text{diff}}, \hat{F}^{2}_{\text{diff}}, \hat{F}^{3}_{\text{diff}}, \hat{F}^{4}_{\text{diff}})\big) eq. 8

\hat{F} = \text{ConvTranspose2D}(S=4, K=3)(F) eq. 9

CM = \text{Linear}(C_{\text{ebd}}, N_{\text{cls}})(\hat{F}) eq. 10

# MLP decoder over the four difference maps  [§2.2]
# diffs[i] has C_i channels at resolution H / 2 ** (i + 1)  [§2.1]
def mlp_decoder(diffs):
    upsampled = []
    for diff in diffs:
        # Per-pixel linear map to the embedding size  [eq. 6]
        projected = linear(channels(diff), embed_dim)(diff)
        # Bilinear resize to H/4 x W/4  [eq. 7]
        upsampled.append(resize(projected, H // 4, W // 4))
    # Concatenate the four maps and fuse them linearly  [eq. 8]
    fused = linear(4 * embed_dim, embed_dim)(concatenate(upsampled))
    # Transposed convolution, stride 4 and kernel 3, to full size  [eq. 9]
    up = conv_transpose(embed_dim, embed_dim, kernel=3, stride=4)(fused)
    # Per-pixel linear classifier: change and no-change logits  [eq. 10]
    return linear(embed_dim, num_classes)(up)  # (B, 2, H, W)

Training step

# One training step  [§3.2]
# Cross-entropy over two classes; AdamW, learning rate 1e-4  [§3.2]
def train_step(model, optimizer, pre, post, label):
    logits = model(pre, post)  # (B, 2, H, W)
    loss = cross_entropy(logits, label)  # [§3.2]
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()  # learning rate decays linearly to 0  [§3.2]
    return loss

Figures

• fig. 1 shows the architecture: the Siamese encoder, the four difference modules and the decoder.
• fig. 2 shows qualitative change maps on LEVIR-CD and DSIFN-CD. The authors say their maps capture finer detail than the other methods, highlighted in red §4. The figures are not reproduced here.

Hyperparameters

Values from the paper unless marked as code. Cells say "not stated" where the paper is silent.

Results

Author-reported. Setup: LEVIR-CD and DSIFN-CD, change-class metrics on the test patches §3.1 §3.3. The table compares eight earlier methods §4 tab. 1.

Values in percent. Source: tab. 1. The caption calls the values averages, but the number of runs and any spread are not stated tab. 1.

• The text reports gains over the previous best, BIT, of about 1.2, 2.2 and 0.1 percent for F1, IoU and OA on LEVIR-CD, and about 20.0, 44.3 and 6.4 percent on DSIFN-CD §4. The LEVIR figures match the relative gains computed from tab. 1. The DSIFN F1 and OA figures do not. tab. 1 gives 17.41 points (25.1 percent relative) for F1 and 6.15 points (6.9 percent relative) for OA. The IoU figure, 23.51 points or 44.4 percent relative, is close to the stated 44.3 percent.
• Precision is mixed: IFNet has the highest LEVIR precision (94.02, above ChangeFormer's 92.05), while ChangeFormer has the highest DSIFN precision (88.48) tab. 1.

Paper vs code

The official repository is github.com/wgcban/ChangeFormer. It is linked from the abstract [abstract]. Its owner, wgcban, is the first author, and the README links his homepage README.md. The code is under the MIT license LICENSE. The commit recorded in this card is afd1b7ed. Every citation below points at that commit.

The training scripts train ChangeFormerV6, which the script comment calls the finalised version run_ChangeFormer_LEVIR.sh. The paper does not name a variant. Each difference below was checked against the code path ChangeFormerV6 uses.

1. Sequence reduction on Q. The paper reduces Q, K and V alike §2.1.1 eq. 2. The code keeps Q at full length ChangeFormer.py and reduces only K and V ChangeFormer.py.
2. Reduction implementation. The paper writes reshape plus linear eq. 2 eq. 3. The code uses a strided convolution with kernel and stride equal to the ratio, followed by LayerNorm ChangeFormer.py ChangeFormer.py.
3. Downsampling kernels in stages 2 to 4. The paper says K=3, S=2, P=1 §2.1.2. ChangeFormerV6 passes patch_size = 7 into the encoder ChangeFormer.py. That value is used for the later patch embeddings, so the code uses K=7, S=2, P=3 there ChangeFormer.py.
4. Positional convolution. The paper writes a 3×3 convolution without groups eq. 4. The code uses a depthwise 3×3 convolution, one filter per channel ChangeFormer.py ChangeFormer.py.
5. Residuals and normalisation. The paper writes no residual around attention and no normalisation in §2.1.1. The code uses pre-norm blocks with residuals around attention and MLP ChangeFormer.py, a LayerNorm after each patch embedding ChangeFormer.py, and a LayerNorm after each stage ChangeFormer.py.
6. Difference module. The paper uses one convolution, ReLU and BN eq. 5. The code uses two convolutions with ReLU and BN between them and a ReLU at the end ChangeFormer.py. The code also projects each map to the embedding size before the difference, whereas the paper applies the MLP projection after it ChangeFormer.py §2.2.1.
7. Top-down accumulation. The paper concatenates the four upsampled maps eq. 8. The code adds the upsampled, coarser fused map to the output of each difference module, from the stage 4 map downwards ChangeFormer.py ChangeFormer.py ChangeFormer.py.
8. Fusion. The paper uses a linear layer eq. 8. The code uses a 1×1 convolution followed by BatchNorm ChangeFormer.py.
9. Upsampling. The paper uses one transposed convolution with stride 4 and kernel 3 eq. 9. The code uses two transposed convolutions with kernel 4 and stride 2, each followed by a residual block ChangeFormer.py ChangeFormerBaseNetworks.py.
10. Classifier. The paper uses a per-pixel linear layer eq. 10. The code uses a 3×3 convolution ChangeFormer.py.
11. Auxiliary change maps. The paper outputs one map §2.2.3. The code predicts a map at each of the four scales as well ChangeFormer.py ChangeFormer.py ChangeFormer.py ChangeFormer.py ChangeFormer.py. The training loss is a weighted sum over the five maps trainer.py with weights from main_cd.py. At inference the final map is used trainer.py.
12. Output activation. The decoder returns logits. A sigmoid is applied only when decoder_softmax is set, which is off by default ChangeFormer.py ChangeFormer.py.
13. Initialisation. The paper says the network is randomly initialised §3.2. The LEVIR script loads pretrained_changeformer.pt run_ChangeFormer_LEVIR.sh, with strict=False trainer.py. The README says the LEVIR training initialises some parameters from a model pre-trained on RGB segmentation with ADE 160k README.md. The DSIFN script starts from the LEVIR best checkpoint run_ChangeFormer_DSIFN.sh.
14. DSIFN learning rate. The paper gives one learning rate, 1e-4 §3.2. The DSIFN script uses 6e-5 run_ChangeFormer_DSIFN.sh.
15. Rescale range. The paper gives 0.8 to 1.2 §3.2. The code's rescale runs from 1.0 to 1.2 data_utils.py, and the training set enables it CD_dataset.py.
16. Metric key. The paper reports change-class F1 and IoU §3.3. The code's training log and checkpoint selection use mf1, the mean F1 over both classes trainer.py metric_tool.py. The per-class F1_1 and iou_1 values are computed too metric_tool.py metric_tool.py. The repository does not say which keys produced Table 1.

Matches: the optimizer, AdamW betas and weight decay §3.2 trainer.py, the batch size and epochs §3.2 run_ChangeFormer_LEVIR.sh, and the patch size §3.1 run_ChangeFormer_LEVIR.sh.

Code-only details the paper omits: normalisation of each channel to [-1, 1] with mean 0.5 and std 0.5 data_utils.py data_utils.py; vertical and horizontal flips with probability 0.5 data_utils.py data_utils.py; Gaussian blur with a random radius, always applied data_utils.py; and colour jitter of 0.3 on brightness, contrast, saturation and hue, always applied data_utils.py.

Pitfalls

• Reducing Q breaks the residual. Eq. 2 reduces Q, K and V, so the attention output would have HW/R tokens. The residual connection needs HW, so the literal reading does not fit. The code keeps the full queries and reduces only keys and values ChangeFormer.py ChangeFormer.py. Use that.
• R is not given. The paper gives R only symbolically. The code's per-stage spatial strides are 8, 4, 2 and 1 ChangeFormer.py. If R counts the tokens merged into one, that is 64, 16, 4 and 1 for the four stages. This is inferred from the code, not stated in the paper.
• §2.1.2 contradicts itself. It describes "a 3×3 Conv2D layer with kernel size K = 7". Use K=7 for stage 1 and K=3 for the rest as the sentence after it says §2.1.2, or follow the code, which uses K=7 in every stage of ChangeFormerV6 (see Paper vs code, item 3).
• Unstated shapes. The paper does not give the head count, the MLP hidden width, the projection layers for Q, K, V and the output, or the output channels of the difference module. The code uses linear projections for Q, K, V ChangeFormer.py ChangeFormer.py and an output projection ChangeFormer.py.
• Transposed convolution with stride 4 and kernel 3. Stride is larger than kernel, so some output pixels receive no contribution from any input position. The paper gives no output padding and does not say how this gap is handled. This is our reading of the layer's arithmetic, not a statement from the paper. The code avoids it by using kernel 4 with stride 2, twice ChangeFormer.py.
• Initialisation changes the result. The paper says the network is randomly initialised §3.2. The released scripts start from pretrained weights, and strict=False silently skips any key that does not match trainer.py. A replication from random initialisation may not reproduce the table. The paper does not say whether its reported numbers used pretraining.
• Learning rate for DSIFN. The paper gives one learning rate §3.2. The DSIFN script uses 6e-5 run_ChangeFormer_DSIFN.sh, and that is the released setting.
• Schedule does not reach zero. The code multiplies the base rate by 1 − epoch/201, starting at epoch 0 networks.py. The final epoch is at a factor of 2/201, about 0.01, not zero. The paper says the rate decays to zero §3.2.
• Auxiliary losses. Both scripts set multi_scale_train=True run_ChangeFormer_LEVIR.sh and multi_scale_infer=False run_ChangeFormer_LEVIR.sh. Training therefore uses the five weighted losses, and inference uses only the final map trainer.py. The paper describes neither the auxiliary losses nor the weights.
• Augmentation is weaker in code than in the text. The paper's rescale is 0.8 to 1.2 §3.2; the code's is 1.0 to 1.2 data_utils.py. The code's blur and colour jitter are applied to every training sample data_utils.py data_utils.py. Implement the code's probabilities or state the choice.
• Metric key. Use change-class F1 and IoU, as the paper states §3.3. The code selects checkpoints on the two-class mean trainer.py. Take the per-class values from metric_tool.py and metric_tool.py.
• Text percentages do not match the table on DSIFN. See Results: the F1 and OA gains quoted in §4 do not follow from Table 1.
• No variance and no compute cost. The table caption says "average", but the number of runs, seeds and spread are not stated tab. 1. Parameter count, FLOPs and training time are not stated.
• Dataset preparation is not described in the paper. The code has MATLAB scripts for patch extraction levir_cd_256.m dsifn_cd_256.m. The paper gives only the patch sizes and split sizes §3.1.
• Qualitative results are described, not reproduced. fig. 2 shows the claimed finer detail, but the card does not reproduce it.

Lineage

• Builds on: the Vision Transformer's patch-based design [vit] (arXiv:2010.11929) and the sequence reduction from Pyramid Vision Transformer [eff_atten] (arXiv:2102.12122), which the paper says it adopts §2.1.1. Scaled dot-product attention comes from "Attention Is All You Need" [original_attention], which has no arXiv id in the paper's bibliography.
• Compared against tab. 1: FC-EF, FC-Siam-Di and FC-Siam-Conc; DTCDSCN; STANet; IFNet; SNUNet; and BIT, "Remote sensing image change detection with transformers" [transformer_cd], which is the closest transformer baseline and keeps a ResNet18 encoder §1.
• Cited as transformer segmentation designs in the introduction: SegFormer (arXiv:2105.15203) [segformer]. The paper does not say it borrows the decoder from SegFormer.