Continental-Scale Building Detection from High Resolution Satellite Imagery

In plain words

Knowing where buildings are, and what shape they have, helps with mapping, planning and disaster response. In many countries that information is missing or out of date. This paper describes a system that finds buildings across a whole continent from high-resolution satellite photos.

The system labels every pixel of an image as "building" or "not building", using a common image-labelling network. Pixels near the edges between two nearby buildings are given extra weight during training, so the network learns to keep them apart. The authors then group the building pixels into separate footprints. Training draws on hand-labelled images, on a large set of unlabelled images that the model labels for itself, and on a pre-training stage that starts from a general image-recognition model.

The authors ran the system over a large part of Africa and published the resulting set of building footprints as the Open Buildings dataset.

TL;DR

• A U-Net with a ResNet-50-v2 encoder and a modified "residual" decoder predicts a building probability per pixel §4 fig. 5.
• Training combines weighted cross-entropy with a focal Tversky term, and emphasises pixels near instance edges via a Gaussian-blurred edge map §5 eq. 1 eq. 3 §5.1.
• Mixup on the cross-entropy term, ImageNet initialisation, and self-training with soft teacher labels and a KL loss are the other main ingredients §6 §7 §8.
• Author-reported: mixup and edge weighting are the largest ablation effects; self-training adds +0.057 mAP on average §11 fig. 8.
• The method was applied to 19.4M km² of imagery and produced 516M building polygons after deduplication §12.

Problem

Building footprints are needed for mapping, population estimates and planning, and matter most where censuses are rare or where settlement changes quickly §1. Prior building detectors were limited to particular cities or countries, and most used one U-Net-style segmentation model §1 §2. Continent-scale work faces varied terrain, vegetation that resembles buildings, touching buildings with no clear boundary, and very small buildings that span only a few pixels at 50 cm per pixel §1. The paper asks how well such a pipeline generalises across Africa §1 §3.1.

Method

Overview. The model is a semantic segmenter, not an end-to-end instance detector. Each pixel gets a building confidence. A threshold and connected-component search then give instances §4 §10. The authors' main contribution is the set of training changes that improve this pipeline, not a new architecture §1 §13.

Steps.

1. Labels. Each labelled training image is 600 × 600 px at 50 cm tab. 1 §3.1. Building instances are eroded by one pixel with a 3 × 3 kernel before training §9. Dense-building regions are relabelled as building during training §9.
2. Network. A U-Net whose encoder is ResNet-50-v2. Deeper encoders (ResNet-v2-101 and -152) were tried in preliminary experiments and did not help §4.
3. Decoder block. Each block has two batch-norm, ReLU, convolution units, then one batch-norm and ReLU unit. A residual connection adds the block input back, and an up-convolution upsamples the result §4 fig. 5.
4. Augmentation. Random 448 × 448 crops from the 600 × 600 image, horizontal and vertical flips, rotations, and random changes to brightness, hue, saturation and contrast §6.
5. Edge-weight map. Build an edge image E that is 1 on instance boundaries and 0 elsewhere. Convolve E with a Gaussian of length scale σ = 3, then multiply by c = 200. The result is the per-pixel weight ω §5.1 fig. 6. This replaces the U-Net distance-based weight, which is slow to compute and forces pre-computed weights §5.1 eq. 4.
6. Loss. Total loss is the weighted cross-entropy plus α = 0.5 times a focal Tversky loss eq. 1 eq. 2 eq. 3. The focal Tversky term is not weighted by ω §5.
7. Mixup. Blend two training images with ratio λ and apply cross-entropy to each of the two label maps, weighted by λ and 1 − λ. The labels are not blended. Mixup is applied only to the cross-entropy term §6 eq. 5 eq. 6.
8. Pre-training. Start from an ImageNet classifier for the encoder. The decoder starts random §7. Satellite-specific pre-training alone (night-time luminance, S2 location classes) scored below ImageNet. Fine-tuning an ImageNet model on luminance raised mAP slightly, not significantly §3.2 §7 tab. 3.
9. Supervised training. Optimiser, learning rate, decay, steps and batch size are in the baseline configuration §11 tab. 2.
10. Self-training. The supervised model is the teacher. The student is trained on unlabelled images with soft teacher labels and a KL loss with focal γ = 0.25 §8. The focal KL form comes from a cited work and is not written out in the paper. Random black masks are added to student inputs so that the student handles black no-data regions §8. Several iterations were run; the number is not stated §8.
11. Unlabelled set. 100M 640 × 640 px images were sampled. The best supervised model filtered them down to 8.7M, keeping only about 1/8 images without buildings §3.3.
12. Inference. Average the confidence maps of the models (ensemble), and average the scales 1, 512/448 and 576/448 §10.
13. Instances. Threshold the averaged confidence. Take 4-connected components. The instance score is the mean confidence of its pixels. Dilate each instance with a 3 × 3 kernel to undo the erosion from step 1 §10 §9.
14. Score thresholds. For each level-4 S2 cell, compute the score thresholds that give 80%, 85% or 90% precision at IoU 0.5, using the sparse additional-evaluation set reweighted to the Open Buildings density §12 fig. 12.

Pseudocode for the weight map, the loss and the training steps:

# Pixel weights from instance edges  [§5.1]
# Input:  labels (H, W) with 1 = building;  sigma = 3;  c = 200
# Output: weights (H, W)

def pixel_weights(labels):
    # 1. Mark pixels on the boundary of each instance  [§5.1, step 1]
    edges = boundary_pixels(labels)          # E(i) = 1 on a boundary
    # 2. Blur the edges with a Gaussian, then scale by c  [§5.1, step 2]
    weights = c * gaussian_convolve(edges, sigma)
    return weights

# Mixed training loss  [§5] [eq. 1] [eq. 2] [eq. 3]
# Input:  probs (B, H, W) building confidence;  labels (B, H, W)
# Output: scalar loss

def mixed_loss(probs, labels, weights):
    # Pixel-weighted cross-entropy  [eq. 1] [§5.1]
    ce = weighted_cross_entropy(probs, labels, weights)
    # Focal Tversky term, weight alpha = 0.5  [eq. 2] [eq. 3]
    ftl = focal_tversky(probs, labels, beta=0.99, gamma=0.25, eps=1e-6)
    return ce + 0.5 * ftl

# Focal Tversky loss  [eq. 2]
# Input:  probs and labels (per image or batch: not stated)
# Output: scalar loss

def focal_tversky(probs, labels, beta, gamma, eps):
    # Overlap term: sum of y * yhat  [eq. 2]
    overlap = sum(labels * probs) + eps
    # Denominator: (1 - beta) on labels, beta on predictions  [eq. 2]
    denominator = sum((1 - beta) * labels + beta * probs) + eps
    return (1 - overlap / denominator) ** gamma

# Mixup on the cross-entropy term  [§6] [eq. 5] [eq. 6]
# lambda = 0.05 in the paper's experiments  [§6]

def mixup_cross_entropy(image_a, labels_a, image_b, labels_b, lam):
    # Blend the two images and predict once  [eq. 5]
    mixed = lam * image_a + (1 - lam) * image_b
    probs = model(mixed)
    # Keep the two label maps separate; no blended label  [eq. 6]
    loss_a = cross_entropy(probs, labels_a)
    loss_b = cross_entropy(probs, labels_b)
    return lam * loss_a + (1 - lam) * loss_b

# Self-training step with soft teacher labels  [§8]
# Input:  unlabelled images;  teacher and student models
# Output: scalar KL loss (focal, gamma = 0.25)

def self_training_loss(student, teacher, images):
    # Soft teacher confidences are the targets  [§8]
    targets = teacher(images)     # input to teacher not stated
    # Random black masks on the student input  [§8]
    masked = random_black_mask(images)
    predictions = student(masked)
    return focal_kl(predictions, targets, gamma=0.25)   # formula not stated

# Inference to building instances  [§10] [§9]
# Input:  image;  models;  threshold chosen per region  [§12]
# Output: list of instance masks

def detect_buildings(image, models, threshold):
    # Average over models and over scales 1, 512/448, 576/448  [§10]
    maps = [model(resize(image, s)) for model in models for s in scales]
    confidence = mean(maps)
    # Threshold, then 4-connected components  [§10]
    mask = confidence >= threshold
    instances = connected_components_4(mask)
    # Dilate each instance by 3 x 3 to undo the erosion  [§10] [§9]
    return [dilate_3x3(instance) for instance in instances]

Hyperparameters

Results

Author-reported. Setup: continent-wide Africa data at 50 cm; U-Net with ResNet-50-v2; metric is mAP at IoU 0.5 with COCO metrics on the 1,920-image test set §3.1 §11.

The final model's absolute mAP on the test set is not stated in the text. Per-category and per-region precision-recall curves are only shown as figures fig. 9 fig. 10.

Paper vs code

No official code for the paper's pipeline. Searched: the paper and its abstract contain no code link. The authors' project page (sites.research.google/open-buildings) links only Colab notebooks in google-research/google-research/building_detection/. That folder holds a README and four notebooks: download and visualisation of Open Buildings polygons and imagery, and one example, satellite_imagery_example.ipynb (read via WebFetch, not cloned). That example uses a subset of SpaceNet Buildings v2 (Khartoum), a small Keras encoder-decoder with residual connections, RMSprop, sparse categorical cross-entropy, batch size 16, and 448 × 448 inputs. It has no focal Tversky loss, no mixup, no self-training, and no edge weighting. It is not the paper's pipeline, so no code field is set. The third-party repository opengeos/open-buildings is not by the authors and was not used.

Pitfalls

• Abstract vs body. The abstract gives mixup +0.12 and self-training +0.06 mAP. The body gives +0.057 for self-training. The mixup figure is only an image in fig. 8, so its absolute baseline and value are not in the text. Use the body numbers and cite the figure.
• Edge weight formula. The paper says ω is "E convolved with a Gaussian, then scaled by c". The kernel size, normalisation, and whether σ is in pixels are not stated. An implementer has to pick these and should record the choice.
• Edge weights apply to cross-entropy only. Focal Tversky is unweighted §5. Do not weight both.
• Focal Tversky reduction. Eq. 2 sums over pixels, but whether the sum is per image, per batch, or whether the mean is used is not stated.
• β direction. By the equation, β multiplies predicted probability and (1 − β) multiplies the label. With β = 0.99 the loss is dominated by predicted mass, which penalises false positives more. This is read from the equation, not stated in prose.
• γ < 1. The paper uses γ = 0.25 although focal losses usually use γ > 1. It reports γ > 1 hurt, and suspects label noise §5. Keep 0.25.
• Mixup weights. Eq. 6 weights the label-a loss by λ and the label-b loss by 1 − λ. With λ = 0.05, the second label map gets 95% of the loss. The paper also says λ ∈ [0, 1) and does not say how partner images are chosen or how λ is sampled §6.
• Mixup and labels. Labels are not blended, unlike original mixup. The authors chose this because pixel weighting made blended labels worse in preliminary runs §6.
• Dense-building label handling. Mapping dense labels to building is the default. Setting their weight to 0 gave equivalent results §9. Either is consistent with the paper.
• Erosion and dilation. Erosion is applied only to labelled training images §9. Dilation is only an approximate inverse §10.
• Self-training KL. The exact KL formula with focal γ is in a cited work and is not written out here §8. The teacher's input (augmented or not) is not stated.
• Self-training masks. Random black masks are applied to the student so it handles no-data pixels. The paper traces an earlier failure to incorrect labels near black regions §8.
• Unlabelled filtering. The threshold used by the supervised model to keep 8.7M of 100M images is not stated §3.3.
• Scoring. The 80/85/90% precision thresholds were estimated on the sparse additional set, reweighted to match Open Buildings density. They depend on the sampling design and do not estimate recall §3.4 §12.
• Test set design. The test set uses 47 chosen regions to avoid bias to empty areas §3.1. Scores on random tiles will differ.
• Model cost. Training cost, inference throughput, and hardware are not stated. The scale is 19.4M km² and 36B polygons before deduplication §12.
• Imagery. Source imagery is internal and at mixed timestamps and resolutions. Orthorectification errors can shift footprints by a few metres §12 fig. 13.
• Annotation. Labels are human-drawn with a "dense building" class. Errors in rural labels and missed buildings are discussed as causes of low scores §11 fig. 10.

Lineage

• Builds on: U-Net, the base architecture §4; ResNet-v2 encoder §4; TernausNetV2, the mixed cross-entropy and Dice loss idea §2 §5; mixup for classification, adapted to segmentation §6; Noisy Student and Naive Student for self-training §8; Focal Tversky and focal KL losses §5 §8.
• Compared against: TernausNetV2 and distance-weighted U-Net variants §2; HRNet and U-Net entries of the SpaceNet challenges §2. The paper reports ablations, not a head-to-head with other architectures §11.
• Superseded by: not stated.