S2Looking: A Satellite Side-Looking Dataset for Building Change Detection

In plain words

Change detection means looking at two photos of the same place, taken at different times, and marking what changed. Here the target is new or removed buildings. Most public training sets for this task use photos taken straight down from above. This paper adds a set taken by satellites that tilt their cameras sideways to photograph a site. Tall buildings then lean and shift between the two photos, so the same building looks different from each side.

The authors collected rural scenes from many countries, aligned each pair of photos as well as they could, and had experts and annotators draw the new and the demolished buildings pixel by pixel. They then ran nine existing change-detection networks on it. All of them did clearly worse than on a comparable straight-down dataset. The point is a harder test and a training resource, not a new model.

TL;DR

• A dataset of 5000 bitemporal RGB pairs of 1024 x 1024 pixels from side-looking satellite cameras, over rural areas worldwide §2.1 tab. 2.
• Pixel-level labels separate newly built buildings from demolished ones §2.2.
• On the authors' benchmark, nine networks score lower on S2Looking than on LEVIR-CD+; the best is BiT with F1 0.6185 tab. 3.
• Use it as a harder test set or training data for building change detection; do not expect a paper method to reuse.

Problem

Most public change-detection datasets use near-nadir (near straight-down) imagery, and their buildings look relatively simple. Side-looking imagery offers larger viewing angles and more useful detail, but no public dataset covers it for building change §1 §2.1. Off-nadir views also cause geometric inconsistency between the two dates, so buildings do not line up pixel by pixel, which needs to be handled by registration and by the models §2.1 fig. 3.

Method

The paper has no model of its own. The "method" is a dataset construction pipeline §2.2 fig. 6 and an evaluation protocol with nine benchmark networks §3.1. Dataset summary, as stated in the paper:

LEVIR-CD+ for comparison: 985 pairs, 1024 x 1024, 0.5 m, 48,455 change instances, 47,802,614 change pixels tab. 1.

Construction, as steps §2.2:

1. Keep only rural locations that were imaged more than once in the past 10 years.
2. For each location, keep bitemporal image files with intersection-over-union above 0.7 and a time gap above 1 year.
3. Cut each intersecting area into 3000 x 3000 slices, stride 1024, using the coordinates derived from satellite position and camera angle (about 20 m accuracy, about 40 to 100 pixels).
4. Register each slice pair with SIFT feature matching on floating-point gray values, reject outliers with RANSAC homography, resample with the homography, and keep the central 1024 x 1024 area.
5. Have experts screen every pair so the maximum registration deviation is under 8 pixels §2.2.
6. Annotate new and demolished buildings as two separate label maps (Label 1 and Label 2). Each sample was annotated by one annotator and checked by a second; batches were reviewed by experts §2.2 fig. 2.
7. Convert the original 16-bit TIFF images to 8-bit PNG §2.2.

# Build S2Looking-style bitemporal pairs  [§2.2] [fig. 6]
# Input:  archive of side-looking images per rural location
# Output: registered 1024 x 1024 pairs with two label maps
def build_pairs(archive):
    # Keep locations imaged more than once in the past 10 years  [§2.2]
    for location in archive.locations():
        # Keep file pairs with IoU > 0.7 and more than 1 year apart  [§2.2]
        for before, after in bitemporal_pairs(location):
            # Cut 3000 x 3000 slices, stride 1024, from geo coordinates  [§2.2]
            for slice_before, slice_after in cut_slices(before, after):
                # Register: SIFT matches, RANSAC homography, resample  [§2.2]
                aligned = register(slice_before, slice_after)
                # Keep the central 1024 x 1024 area  [§2.2]
                center = crop_center(aligned, size=1024)
                # Expert screening: deviation under 8 px  [§2.2]
                # Annotate new and demolished buildings as two maps  [§2.2]
                labels = annotate(center)
                yield center, labels

Benchmark protocol §3.1. Inputs are a bitemporal image pair; output is a single-channel change map. Scoring is pixel-level precision, recall and F1 eq. 1 eq. 2 eq. 3:

# Pixel-level scoring of one predicted change map  [§3.1] [eq. 1-3]
# Input:  prediction, label: (H, W) binary, 1 = changed (assumed)
# Output: precision, recall, f1
def score(prediction, label):
    # Count true positives, false positives and false negatives  [§3.1]
    true_positive = sum(prediction AND label)
    false_positive = sum(prediction AND NOT label)
    false_negative = sum(NOT prediction AND label)
    # Precision = TP / (TP + FP)  [eq. 1]
    precision = true_positive / (true_positive + false_positive)
    # Recall = TP / (TP + FN)  [eq. 2]
    recall = true_positive / (true_positive + false_negative)
    # F1 is the harmonic mean of precision and recall  [eq. 3]
    f1 = 2 / (1 / precision + 1 / recall)
    return precision, recall, f1

Figure content, described in words: fig. 1 shows scroll imaging; fig. 2 gives sample pairs with their label maps; fig. 3 shows near-nadir and two side-looking views with a visible shift of tall objects; fig. 4 maps the locations, mostly rural areas near marked cities; fig. 5 is the histogram of off-nadir angles, peaking between -5 and 0 degrees §2.1; fig. 7 shows matched feature points; fig. 8 plots precision, recall and F1 per training iteration for both datasets; fig. 9 shows visual predictions of the methods.

Hyperparameters

Results

Author-reported. Setup: each of nine networks (four classic, five state-of-the-art) fine-tuned on LEVIR-CD+ and on S2Looking, scored with pixel precision, recall and F1 on the test split §3.1 tab. 3. Hardware and framework as in the Hyperparameters section.

On-nadir subset of S2Looking (angles within plus or minus 15 degrees, 3597 pairs) tab. 4:

The paper's headline: every method scores lower on S2Looking than on LEVIR-CD+, and the authors report F1 drops of at least 25% §3.2. BiT is the best method on both datasets §4. On the on-nadir subset the methods score much higher §3.2 tab. 4.

Paper vs code

No official code. The paper's data link points to github.com/S2Looking/. I checked that account. Its Dataset repository holds data only: a README with download links, no code and no LICENSE. Its other repositories (CADOR-Net) are a different project. The benchmark networks are not described by source repository; the paper says only that "default settings" of each method were used §3.1. I did not run a separate GitHub search for benchmark code.

Pitfalls

• This is a dataset and benchmark, not a method. Nothing here is a model to implement. The benchmark networks are third-party, and the card names them only as they appear in tab. 3.
• Input modality is contradictory. Section 3.1 says "three-channel multispectral image pairs" §3.1, while Table 2 says "RGB" tab. 2. Confirm the bands from the data before you rely on either.
• Off-nadir range is contradictory. The text gives angles from -35 to +40 degrees §2.1, but the table gives a maximum absolute value of 35.370 degrees tab. 2. The sign convention is not stated, so the +40 side is unreconciled.
• The on-nadir subset is described in two ways. The text says "a subset of the pixels" §3.2, while the same paragraph says the subset "contained 3597 image pairs". Which unit was evaluated is ambiguous.
• Aggregation of scores is not stated. Eq. 1 to 3 do not say whether TP, FP and FN are summed over the test set or averaged per image. The class is written as "class 0 or 1" §3.1; that the changed class is positive is assumed here.
• The label merge is not stated. The text says Label 1 and Label 2 "combined into one map" §3.1. The combination rule (a union is implied) is not written.
• The "at least 25%" claim does not hold for every method. The paper says the F1 scores on S2Looking were "at least 25% lower" §3.2. From tab. 3, CDNet falls from 0.8046 to 0.6056, which is 24.7% relative (computed). The smallest absolute drop is 19.9 points for CDNet (computed).
• The 46.4% on-nadir figure depends on how it is computed. It reproduces as the mean of nine per-method ratios of on-nadir F1 to overall S2Looking F1, using tab. 4 and tab. 3 (computed). The ratio of the two mean F1 scores is about 31% (computed). Check which one you quote.
• The dataset comparison mixes protocols. LEVIR-CD+ uses a 65/35 split with no validation set; S2Looking uses 70/10/20 §3.1. Classic methods get 1024 x 1024 inputs and state-of-the-art methods get 256 x 256 crops §3.1. The "N" iteration unit is undefined. The gap between datasets therefore mixes data and protocol effects.
• The training recipe is missing. Optimizer, learning rate, batch size, loss weights and epoch count are not stated. Results depend on the "default settings" of each third-party method, which are not named §3.1.
• Registration is partly manual, and its method is not reproducible. SIFT and RANSAC parameters are not given §2.2. Expert screening, not an automatic check, bounds the misalignment to 8 pixels. The paper also relates the leftover misaligned area to the average building change size, given as 1056 pixels, and puts it at under one sixteenth of that size §2.2; the unit of "1056 pixels" (area or length) is unclear.
• Slices overlap heavily. 3000-pixel slices with stride 1024 overlap by 1976 pixels (computed), so one scene yields many overlapping candidate crops §2.2.
• Bit depth is lost at conversion. Images were converted from 16-bit TIFF to 8-bit PNG, and the method is not stated §2.2.
• Geographic information is removed for the challenge. Coordinates are removed from the inference data, and no geographic base map may be used §2.4. Do not use geo features if you follow the challenge rules.
• Data licence is not stated. The paper gives no dataset licence, and the Dataset repository has no LICENSE. The card covers only the paper (CC BY 4.0). Do not redistribute the data.
• Version check not done. The LaTeX source was used. The arXiv record is v3, dated 2022-01-11, and we did not compare v3 with the source.
• Annotation cost. About 2000 person-days were spent on annotation and review §2.2.

Lineage

• Builds on: LEVIR-CD and LEVIR-CD+ (Chen and Shi, 2020) as the closest comparison dataset §2.1 §3.1. SIFT §2.2 and RANSAC homography §2.2 are used for registration.
• Compared against: FC-EF, FC-Siam-Conc, FC-Siam-Diff, DTCDSCN, STANet-Base, STANet-BAM, STANet-PAM, CDNet and BiT §3.1 tab. 3.
• Superseded by: not known from content in this wiki.