wikillm

Paper · 2022-01-24 · cs.CV, eess.IV

SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal

Introduces SEN12MS-CR-TS, a paired Sentinel-1 and Sentinel-2 time-series dataset for cloud removal over 53 regions. Two models use it: a 3D-convolution network fusing three cloudy optical dates with radar, and an internal-learning model that cleans a series without external training. Best sequence-to-point NRMSE is 0.051, against 0.079 for the least-cloudy baseline. Author-reported.

Image inpaintingMultimodal fusionInternal learningSiamese networkremote-sensingcloud-removalsar-optical-fusionimage-reconstructiontime-seriesdataset

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Clouds block the view of a satellite's optical camera, so many of its pictures of the ground have holes where the clouds are. The same satellite family also carries a radar, which is less affected by the atmosphere. This paper asks how well one can fill the holes when a place is observed many times over a year, with both optical and radar pictures.

The authors built a dataset for that task. It has many places around the world, each observed by radar and by the optical camera on many dates across one year. An automatic detector gives each optical picture a cloud map, so a model can be trained and scored on the real cloud pattern.

They then test two ways of using it. The first looks at several cloudy optical pictures from different dates, together with the radar pictures, and produces one cloud-free picture of the target date. The second takes a whole cloudy series and rebuilds a cloud-free series, without any outside training data, by fitting a network to that one series. The first beats simple fixes such as copying the least cloudy picture on most measures. The second beats the earlier low-rank baselines it was compared with. These numbers come from the authors' own experiments.

TL;DR

  • SEN12MS-CR-TS pairs radar and optical satellite images of 53 regions over one year, with cloud masks computed for each optical image §II §II.C.
  • Sequence-to-point model: a shared ResNet branch per input date, followed by 3D convolutions, trained with an L1 loss plus a VGG16 perceptual loss §III.A eq. 1.
  • Sequence-to-sequence model: an encoder-decoder trained from scratch on each target series (internal learning), with a masked loss over cloud-free pixels §III.B alg. 1 eq. 4.
  • Headline numbers are author-reported: the best sequence-to-point NRMSE is 0.051 against 0.079 for the least-cloudy baseline tab. I.
  • The repository has code for the sequence-to-point model and the data loader but no LICENSE file, and no seq2seq training code matching the paper (see Paper vs code).

Problem

Clouds cover a large share of optical satellite observations, and this blocks continuous monitoring of the surface §I. Earlier cloud-removal data sets were mono-temporal: one date per place, or cloud patterns that were simulated rather than natural §I.A.

The paper argues that a useful data set needs three things together: radar, which is more robust to atmospheric disturbance §I; repeated observations of the same place to fill holes across time; and the full range of natural cloud coverage. Earlier time-series work used three RGB or near-infrared dates and excluded scenes with more than 30 percent cloud §I.A.

The task has two regimes. In sequence-to-point, a cloud-free image of one date is predicted from a cloudy series. In sequence-to-sequence, a whole cloudy series is cleaned, with no cloud-free training data §III.

Method

Dataset §II

  • 53 regions of interest (ROI), each over 4000 × 4000 px and about 40 × 40 km. 40 ROI form the training split and 13 the held-out test split §II.
  • For each ROI, 30 co-registered Sentinel-1 (SAR) and Sentinel-2 (optical) observations, evenly spaced over the year 2018 §II §II.A. Paired S1 and S2 observations lie within a two-week window §II.A.
  • The mean time difference between paired observations is 2.61 with standard deviation 2.41; the units are not stated app. A fig. 12.
  • Optical data are Sentinel-2 level-1C top-of-atmosphere reflectance, and data were acquired through Google Earth Engine §II.A.
  • Cloud coverage is kept across the full range; the histogram means are about 44 % (±42 %) for train and about 50 % (±42 %) for test fig. 4 fig. 5 §II.
  • The data set is about 2 TB §II.

Preprocessing §II.B

  1. Every band is upsampled to 10 m, the native resolution of Sentinel-2 bands 2, 3, 4 and 8 §II.B.
  2. Each full-scene image is cut into non-overlapping patches of 256 × 256 px §II.B.
  3. Sentinel-1 scenes go through the Sentinel-1 toolbox. Edge and thermal noise are removed, values are calibrated and geometrically corrected, and the result is converted to decibels §II.B.
  4. For ResNet-based models, S1 VV and VH are clipped to [-25, 0] and [-32.5, 0] and mapped to [0, 2]. S2 is clipped to [0, 10000] and mapped to [0, 5] §II.B.
  5. For other backbones, each patch is clipped and rescaled to [0, 1] with a per-pixel rule as stated in the paper §II.B. The code uses a global range instead; see Paper vs code.
  6. For evaluation, all inputs, targets and predictions are mapped to [0, 1] before metrics are computed §II.B.

Cloud masks §II.C

Each optical image gets a binary cloud mask from the s2cloudless detector. The masks are computed on the fly, not stored §II.C. The paper says s2cloudless is about as accurate as the multi-temporal MAJA classifier and cheaper to run §II.C. It gives no detector threshold; the code uses 0.4 dataLoader.py.

Sequence-to-point cloud removal §III.A

Input is a series of n tuples (S1, S2) at times t = 1 … n. Output is one cloud-free S2 prediction §III.A fig. 7. The model has three parts:

  • Each time step goes through its own copy of a ResNet branch, with weights shared across time (Siamese). The branch is the ResNet cloud-removal network of Meraner et al. §III.A.
  • The branch outputs are stacked along the time axis and fused with 3D convolutions into one image §III.A fig. 7.
  • The branch is pretrained on SEN12MS-CR; its weights stay frozen for the first 25000 steps §IV.C.

Loss, with λ_L1 = 100 and λ_perc = 1 eq. 1 §IV.C:

  • L_all = λ_L1 · L_L1 + λ_perc · L_perc eq. 1.
  • L_L1 is the L1 norm between S2 and the prediction eq. 2.
  • L_perc compares VGG16 features of S2 and the prediction eq. 3. The VGG16 is pretrained for land-cover classification on SEN12MS, following a protocol from Schmitt et al. §IV.C.
Pseudocode
# Sequence-to-point forward pass  [§III.A] [fig. 7]
# Input:  s2_steps (n, B, 13, H, W); s1_steps (n, B, 2, H, W)
# Output: cloud-free S2 prediction (B, 13, H, W), values in [0, 5]

def predict_cloud_free(s2_steps, s1_steps):
    # Run the same pretrained ResNet branch on each date  [§III.A]
    branch_outputs = []
    for t in range(n):
        # Concatenate optical and radar channels for date t  [§III.A]
        inputs = concatenate(s2_steps[t], s1_steps[t])  # (B, 15, H, W)
        branch_outputs.append(resnet_branch(inputs))    # (B, 256, H, W)
    # Stack the dates on a time axis and fuse them with 3D convolutions  [§III.A]
    volume = stack(branch_outputs, axis=time)  # (B, 256, n, H, W)
    fused = conv3d_fusion(volume)              # time axis collapses to 1
    # Tanh output in [-1, 1]  [code: models/networks_branched.py#L736-L741]
    prediction = tanh_head(fused)               # (B, 13, H, W) in [-1, 1]
    # Map to the [0, 5] range  [code: models/temporal_branched_model.py#L201-L202]
    return 5 * (prediction + 1) / 2
Pseudocode
# Training loss for one sequence-to-point sample  [eq. 1] [eq. 2] [eq. 3]
# Weights: lambda_l1 = 100, lambda_perc = 1  [§IV.C]

def sequence_to_point_loss(prediction, target):
    # L1 term between the prediction and the cloud-free target  [eq. 2]
    l1_term = absolute_difference(prediction, target)
    # Perceptual term on VGG16 features of both images  [eq. 3]
    perceptual_term = vgg16_feature_distance(prediction, target)
    return 100.0 * l1_term + 1.0 * perceptual_term

Training data: S2 input patches are limited to 0–50 % cloud coverage. The target is the least cloudy S2 patch of the series, which need not be the nearest in time. Each location of each training ROI contributes one tuple, for 10 epochs. The optimiser is Adam with learning rate 0.0002 and betas [0.5, 0.999], batch size 1 §IV.C.

Sequence-to-sequence cloud removal §III.B

The network maps a series of S1 images to a series of cloud-free S2 images. It is a 3D encoder-decoder with U-Net-style skip connections §III.B fig. 8. For ablation, the input can be Gaussian noise instead of S1 §III.B fig. 8.

Training is internal learning: a new network is trained from scratch on the target series, with no external data §III.B. The network first fits the regular structure of the series and then starts to fit the cloudy pixels, so training is stopped before that point §III.B alg. 1.

The loss uses only cloud-free pixels: L_all = λ_L2 · L_L2 + λ_perc · L_perc eq. 4, with masked L2 eq. 5 and masked VGG16 perceptual terms eq. 6, λ_L2 = 1 and λ_perc = 0.01 §IV.D.

Pseudocode
# Internal learning for cloud removal  [alg. 1] [§III.B]
# Input:  s1_series (T, 2, H, W); s2_target (T, 13, H, W); cloud_mask (T, 1, H, W)
# Output: predicted cloud-free S2 series (T, 13, H, W)

def remove_clouds(s1_series, s2_target, cloud_mask, total_iterations):
    # Start from a freshly initialised network, with no pretraining  [§III.B]
    network = new_encoder_decoder_3d()
    for iteration in range(total_iterations):
        # Predict the whole series from the S1 input  [§III.B] [alg. 1]
        s2_predicted = network(s1_series)
        # Score only cloud-free pixels against the cloudy target  [eq. 4] [eq. 5]
        loss = masked_loss(s2_target, s2_predicted, cloud_mask)
        network.backpropagate(loss)
    return s2_predicted

Training runs for 20 passes, each of 100 iterations, with batches of 5 temporally adjacent samples. Optimiser: Adam with learning rate 0.01 §IV.D. The paper says the hyperparameters of the algorithm follow Zhang et al. §IV.D, which are not restated here.

Evaluation for seq2seq: the least cloudy S2 observation is the target, and the most cloudy one is the source. The cloudy pixels of the source are mixed into the clear pixels of the target. The prediction is compared with the original target §IV.D.

Metrics §IV.A

With pixel values in [0, 1], the paper defines NRMSE (root mean squared error over C · H · W values), PSNR as 20 · log10(1 / NRMSE), SSIM, and the spectral angle mapper (SAM). The text gives C = 3 for the metric dimensions (see Pitfalls). The paper reports NRMSE over all pixels, over cloud-covered pixels and over cloud-free pixels §IV.A.

Hyperparameters

Values stated in the paper, with the code's values where they differ. Blank paper values are "not stated".

NamePaper valueWhereCode / note
Input dates n (seq2point)3 (4 and 5 tested)§IV.C tab. IVdefault 3 base_options.py
Input cloud coverage0–50 %§IV.Cdefault 0–100 %; filter only when --max_cov is passed base_options.py README.md
Targetleast cloudy S2 patch§IV.CdataLoader.py
Epochs10§IV.Cdefaults are 100 + 100 epochs; README does not override train_options.py
Batch size1 tuple§IV.Cforced to 1 train.py
OptimiserAdam§IV.Ctemporal_branched_model.py
Learning rate0.0002§IV.Ctrain_options.py
Adam betas[0.5, 0.999]§IV.Ctrain_options.py temporal_branched_model.py
λ_L1100§IV.C eq. 1temporal_branched_model.py
λ_perc1§IV.C eq. 1base_options.py
Frozen steps (pretrained branch)25000§IV.Cdefault 100000 base_options.py
Pretrained branchResNet of Meraner et al., pretrained on SEN12MS-CR§IV.Cweights are external, not in repository networks_branched.py
Branch width and depthnot stated§III.AF = 256, 16 residual blocks network_resnet_branched.py
Perceptual networkVGG16, landcover-pretrained on SEN12MS§IV.C13 input bands, 10 classes, weights from vgg16_path util.py util.py
Perceptual layersnot stated§IV.Cdefault original = layers 3, 8, 15, 22 temporal_branched_model.py
3D fusionnot stated§III.AConv3d 256→256, kernel 3×3×3; five residual 3D blocks (scale 0.1); final Conv3d kernel (n, 3, 3) networks_branched.py
Learning-rate schedulenot stated§IV.Cconstant, then linear decay to zero networks_branched.py
Cloud detectors2cloudless, threshold not stated§II.Cthreshold 0.4, average over 4, dilation 2 dataLoader.py
S2 scaling (ResNet)[0, 10000] → [0, 5]§II.Bdivide by 2000 after clipping dataLoader.py
S1 scaling (ResNet)VV [-25, 0], VH [-32.5, 0] → [0, 2]§II.BdataLoader.py
Seq2seq passes20§IV.Dnone in code (see Paper vs code)
Seq2seq iterations per pass100§IV.Dtotal not stated explicitly
Seq2seq batch5 temporally adjacent samples§IV.Dnone in code
Seq2seq optimiserAdam, learning rate 0.01§IV.Dnone in code
λ_L2 (seq2seq)1§IV.D eq. 4none in code
λ_perc (seq2seq)0.01§IV.D eq. 4none in code
Seq2seq network widthsnot stated§III.B fig. 8from Zhang et al. §III.B

Results

Author-reported. Setup: held-out test split of the data set §II, sequence-to-point model with n = 3, S1 included and perceptual loss on, unless stated §IV.C. Metrics as in §IV.A.

Table I. Sequence-to-point against baselines tab. I:

ModelNRMSE, allNRMSE, cloud-coveredNRMSE, cloud-freePSNRSSIMSAM
least cloudy0.0790.0820.031not reported0.8150.213
mosaicing0.0620.0640.03631.680.8110.250
ResNet0.0600.0620.04026.040.8100.212
STGAN0.0570.0590.05025.420.8180.219
proposed, n = 30.0510.0520.04026.680.8360.186

Table II. Effect of the radar input tab. II:

ModelNRMSE, allNRMSE, cloud-coveredNRMSE, cloud-freePSNRSSIMSAM
without S10.0540.0570.05425.350.8320.194
with S10.0510.0520.04026.680.8360.186

Table III. Effect of the perceptual loss tab. III:

ModelNRMSE, allNRMSE, cloud-coveredNRMSE, cloud-freePSNRSSIMSAM
without perceptual loss0.0520.0530.03926.660.8350.180
with perceptual loss0.0510.0520.04026.680.8360.186

Table IV. Number of input dates tab. IV (the caption says sequence-to-sequence; the values and text match sequence-to-point, see Pitfalls):

ModelNRMSE, allNRMSE, cloud-coveredNRMSE, cloud-freePSNRSSIMSAM
n = 30.0510.0520.04026.680.8360.186
n = 40.0490.0500.04127.100.8450.172
n = 50.0480.0480.03227.070.8460.178

Table V. Performance by input cloud coverage, n = 3 with S1 and perceptual loss tab. V:

Coverage of inputsNRMSE, allNRMSE, cloud-coveredNRMSE, cloud-freePSNRSSIMSAM
0–10 %0.0410.0460.04128.590.8700.143
10–20 %0.0440.0460.04327.690.8480.166
20–30 %0.0460.0470.04427.250.8410.169
30–40 %0.0480.0500.04526.770.8300.169
40–50 %0.0470.0480.04526.860.8300.167
50–60 %0.0490.494 (as printed)0.04826.550.8250.185
60–70 %0.0520.0520.04326.100.8170.184
70–80 %0.0490.0500.04426.590.8160.179
80–90 %0.0500.0500.04426.540.8200.175
90–100 %0.0630.063not reported24.790.7860.222

Table VI. Sequence-to-sequence against baselines tab. VI (the caption calls the first column RMSE, the header says NRMSE):

ModelNRMSE, allPSNRSSIMSAM
RPCP0.4037.9110.26430.567
NMFISL0.31210.2620.45029.285
PNMF0.31710.1350.43229.801
MNMF0.3618.9450.36128.685
OSTD0.30310.8530.40235.454
seq2seq without S10.29811.4340.49428.127
seq2seq with S10.27411.5900.51227.733

The text reports that the proposed sequence-to-point model is best on most metrics: mosaicing is best on PSNR and least-cloudy on NRMSE for cloud-free pixels §IV.C tab. I. For seq2seq, the S1-conditioned model is best on every column of Table VI §IV.D tab. VI.

Paper vs code

The repository is PatrickTUM/SEN12MS-CR-TS (commit 2af9334), under the authors' GitHub account. Its README says it contains code accompanying this paper README.md. The paper itself names no code repository. The authors' project page links only UnCRtainTS, which is a different model, so this card's code link rests on the README. The repository contains a data loader, a sequence-to-point model and test code. Its file listing shows no LICENSE file, so no code is quoted here.

  • Seq2seq code is not in the repository. The ir models in models/ use a GAN plus L1 loss with λ = 100 temporal_ir_model.py temporal_branched_ir_model.py. They have no masked L2 loss, no masked VGG16 term and no 20 × 100 schedule. The paper's seq2seq hyperparameters therefore cannot be checked against code.
  • Perceptual loss. The paper uses the L2 norm eq. 3. The code takes the mean squared error per VGG layer and sums over four layers. On the ResNet path it first rescales the inputs from [0, 5] to [0, 1] temporal_branched_model.py.
  • L1 term. The paper writes the L1 norm eq. 2. The code uses nn.L1Loss, which is the mean over values, times λ_L1 = 100 temporal_branched_model.py. With the ResNet path the L1 term is on the [0, 5] scale and the perceptual term on [0, 1].
  • Frozen steps. The paper says 25000 steps §IV.C. The default is unfreeze_iter = 100000, and the README command does not change it base_options.py README.md.
  • Epochs. The paper says 10 epochs §IV.C. The defaults are 100 constant epochs plus 100 decay epochs, and the README command does not override them train_options.py README.md.
  • Learning-rate schedule. Not in the paper. The code holds the rate for niter epochs and then decays it linearly over niter_decay epochs networks_branched.py.
  • Input coverage. The paper filters inputs to 0–50 % §IV.C. The code applies the filter only when --max_cov is given, and the README command does not give it base_options.py README.md. When fewer than n inputs match, the code takes the first n dates in time order, not a random draw dataLoader.py.
  • Perceptual loss and GAN in the example command. The paper's loss has a perceptual term and no GAN term eq. 1. The README command sets --lambda_GAN 0.0 (the code default is 1.0) and does not set --use_perceptual_loss README.md temporal_branched_model.py base_options.py. Using the README command therefore trains a different loss from the paper.
  • Per-pixel normalisation. The paper says non-ResNet backbones rescale each patch per pixel §II.B. The code rescales with a fixed global range [0, 10000] → [0, 1] dataLoader.py.
  • Metric channels. The paper gives C = 3 §IV.A. The metrics use every output channel, 13 bands test.py.
  • Cloudy and cloud-free sets. The paper defines cloud-covered as cloudy in no input, and cloud-free as visible in at least one input §IV.A. The code marks a pixel cloudy if it is cloudy in at least one input, and cloud-free only if it is clear in all inputs test.py.
  • SAM. The paper gives one spectral angle over the whole image §IV.A. The code computes an angle per pixel and averages them test.py.
  • Mosaicing fallback. The paper uses 0.5 for pixels with no clear observation §IV.B. The code sets 0.5 after rescaling, so the effective value is 0.1 on [0, 1] for ResNet preprocessing and 0.75 on the default path test.py.
  • Pretrained weights. The initial ResNet branch and the 13-band VGG16 are loaded from external files (--initial_model_path, --vgg16_path). Neither is in the repository networks_branched.py util.py.
  • Consistent. The paper's Siamese branches share weights, and the code applies one branch module to each input in turn networks_branched.py.

Pitfalls

  • Scale of the loss terms. The L1 term, with λ_L1 = 100, is computed on [0, 5] values for the ResNet branch, while the perceptual term is on [0, 1] after division by 5 temporal_branched_model.py. The paper does not say which scale the weights assume.
  • Reduction of the L1 term. Norm versus mean changes the effective weight by a factor equal to the number of values. The code uses the mean temporal_branched_model.py.
  • Metric scaling. NRMSE, PSNR and SAM are computed after the rescale to [0, 1], so values depend on the preprocessing branch. The ResNet path divides by 5 test.py. Do not compare numbers across branches.
  • Table caption mismatch. Table IV's caption says sequence-to-sequence, but its values match the sequence-to-point n = 3 row, and the text places it in the sequence-to-point section tab. IV §IV.C.
  • Likely typo in Table V. The 50–60 % row lists 0.494 as NRMSE for cloud-covered pixels. Neighbouring rows are near 0.05. Check against the data before using it tab. V.
  • Table VI labelling. The caption says RMSE, the header says NRMSE tab. VI.
  • Figure cross-reference. The seq2seq text points to Fig. 1 for its example outputs, but the Fig. 1 caption does not name the model §IV.D fig. 1.
  • Pass definition. The paper says 20 passes of 100 iterations each §IV.D. Whether a pass is a full epoch over the target series is not stated.
  • Cloud detection. Masks are recomputed on each read, which costs run time §II.C. The threshold, the averaging and the dilation are only in code dataLoader.py.
  • Size and access. The data are about 2 TB §II. Download is sharded by modality and region, and the README gives the split archives.
  • Time-difference units. The paired time difference is given as 2.61 ± 2.41 without units app. A.
  • Minimum input count. The 3D generator asserts n ≥ 3 networks_branched.py.

Lineage

  • Builds on: the STGAN generator of Sarukkai et al. (the sequence-to-point design, arXiv 1912.06838 per the code README) §III.A README.md. The ResNet branch comes from Meraner et al. §III.A.
  • Builds on, seq2seq: internal learning of Ulyanov et al. (deep image prior) and the 3D encoder-decoder of Zhang et al. §III.B. See deep image prior.
  • Compared against: least cloudy, mosaicing, a ResNet baseline and STGAN (Table I); RPCP, NMFISL, PNMF, MNMF and OSTD (Table VI) tab. I tab. VI.
  • Predecessor data set: SEN12MS-CR, mono-temporal and compatible in its ROI split §V.
  • Superseded by: not stated in this card.

Source and license

Adapted from “SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal” by Patrick Ebel, Yajin Xu, Michael Schmitt, Xiao Xiang Zhu (arXiv:2201.09613), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/PatrickTUM/SEN12MS-CR-TS at commit 2af933422f58 (none (no LICENSE file in repository)). Any quoted code is shown under that license.