Learning from synthetic photorealistic raindrop for single image raindrop removal

In plain words

Water drops that sit on a camera lens or a windscreen blur and hide parts of the scene. Cars with cameras behind the windscreen are a typical case. Learning to remove them needs pairs of images: the same scene with and without drops. Real pairs are hard to collect, so the authors build them. They put virtual drops on clean street photos and simulate how light bends through each drop, using the drop's shape and the scene behind it.

The removal model works in three steps. First it finds where the drops are. Then it repaints those spots from the surrounding picture. Last, a small network smooths the seams. The authors report that it beats the methods they compare against by a wide margin on their own synthetic test set, and that it is on par with the best earlier method on an existing real-image set.

TL;DR

The paper contributes a large synthetic dataset of adherent raindrops rendered on Cityscapes images, with exact drop masks, and a three-part network that detects, reconstructs and refines. On the synthetic test set the full model reaches 41.29 dB PSNR and 0.9921 SSIM (author-reported), against 37.79 dB for Qian et al. On the real-image set from Qian et al. it is roughly on par with that method. No code is released. Relevant if you need paired training data for raindrop removal, but the rendering stage has to be rebuilt from the text.

Problem

Raindrops on a lens or windscreen obstruct, deform and blur the background, and they hurt downstream vision tasks such as detection, tracking and stereo §1. Removing them from a single image is hard for three stated reasons: the drop appearance depends on many parameters, model-based and classical learning methods do not capture it well, and deep learning lacks large, realistic paired data §1 §3. The only earlier paired adherent-raindrop dataset cited has 1119 pairs and no pixel masks §3.

Method

Imagery model. The observed image is written as O = (1 − M)B + R, where M is the raindrop binary mask, B the clear background and R the raindrop layer §4. Section 4 then uses the raindrop image R itself in its equations (see Pitfalls).

Dataset: geometric rendering and ray tracing §3. A camera sits at the origin. A glass plane lies N cm in front of it and a background plane T cm in front. The glass is tilted at angle ψ to the ground. Drops are spherical caps of sphere radius r, with tangent angle τ to the glass; r and τ set the drop volume. Glass refraction is ignored. For each camera pixel the ray is traced. If it misses all drops, the pixel keeps its background value. If it passes through a drop, the ray is refracted, and the pixel takes the value at the point where the traced ray meets the background plane. If total internal reflection occurs on leaving the drop, the pixel is set to black, the "dark band" effect §3.

Dataset: blur and mask §3. The drop regions are blurred with a disk kernel of diameter 7 to 20 pixels. The drop positions are known from the simulation, so the binary mask comes for free §3. Figure 2 shows the refraction geometry fig. 2; Figure 3 shows a clean Cityscapes image, its synthetic counterpart and the mask fig. 3.

Dataset: sampling §3. Parameters are drawn from N in [20, 40], T in [800, 1500], r in [0.8, 1.5], ψ in [30°, 45°] and τ in [30°, 45°]. Each background gets 50 to 70 drops. The training split has about 30,000 images from the Cityscapes training set; the test split has 1,525 rainy/clear pairs from the Cityscapes test set §3 §5.1.

Network. Three sub-networks, trained in two stages §4.

1. Raindrop detection network §4.1. It has five convolution layers and six bottleneck residual blocks, with BN and ReLU after every convolution. The second convolution has stride 2, which halves the resolution. A 1×1 layer raises the channels from 64 to 256. Each bottleneck is a 1×1 reducing to 64, a 3×3 at 64 channels, and a 1×1 expanding to 256 §4.1. Output is a per-pixel mask trained with binary cross-entropy, Eq. 1 eq. 1. Because the mask has little semantic information, the authors keep the feature maps at half size to enlarge the receptive field §4.1.

2. Raindrop region reconstruction network §4.2. It has the same design with eight residual blocks. Its input is the raindrop image plus an edge map, so it is 4-channel. The edge map is the average over the four neighbour differences of the absolute colour change per RGB channel, Eq. 2 eq. 2. Its loss is an L1 loss over all pixels, where pixels inside a raindrop are weighted by 20 and all others by 1 eq. 3 §4.2.

3. Refine network §4.3. The blended input is B = M̂·Î + (1 − M̂)·R eq. 4, where M̂ is the detector's mask, Î the reconstruction and R the raindrop image. Before blending, M̂ is dilated with a parameter-free layer to remove drop edge pixels that the detector misses §4.3 fig. 5. The network has two convolutions and two residual blocks. Its loss mixes an SSIM term and an L1 term with weight α = 0.3, Eq. 5 eq. 5.

Training §4.4. Stage 1 trains the detection and reconstruction networks separately, since the refine network depends on their outputs. Stage 2 freezes them and updates only the refine network. Augmentation: a training pair or its horizontal flip is cropped at random to 100×100 from 256×512 images §4.4. Optimiser: Adam, with learning rate 1e-3 decayed linearly from epoch 20 to epoch 40 down to 1e-4; batch size 32; 50 epochs; Xavier initialisation §4.4.

Pseudocode for inference, following §4 and §4.3:

# Inference pipeline  [§4] [eq. 2] [eq. 4] [§4.3]
# Input:  raindrop_image (H, W, 3)
# Output: derained image (H, W, 3)

def remove_raindrops(raindrop_image):
    # 1. Detect raindrops: per-pixel probability  [§4.1]
    mask_prob = detection_net(raindrop_image)      # (H, W, 1)
    # 2. Binarise and dilate the mask (threshold not stated)  [§4.3] [fig. 5]
    mask = dilate(binarise(mask_prob))             # (H, W, 1)
    # 3. Edge map of the raindrop image  [eq. 2]
    edges = edge_map(raindrop_image)               # (H, W, 1)
    # 4. Reconstruct the content under the drops  [§4.2]
    recon_input = concat(raindrop_image, edges)    # (H, W, 4)
    reconstructed = reconstruction_net(recon_input)  # (H, W, 3)
    # 5. Blend reconstructed pixels into the drop-free area  [eq. 4]
    blended = mask * reconstructed + (1 - mask) * raindrop_image
    # 6. Smooth the blended image  [§4.3]
    return refine_net(blended)

# Losses, as written in the paper  [eq. 3] [eq. 5]
# Gt mask is assumed to be the ground-truth drop mask (not named in [§4.2])

def reconstruction_loss(clean, predicted, gt_mask):
    # Weight 20 on raindrop pixels, 1 elsewhere  [§4.2] [eq. 3]
    weights = where(gt_mask, 20.0, 1.0)
    return mean(weights * abs(clean - predicted))

def refine_loss(clean, refined):
    # alpha = 0.3 on the SSIM term  [§4.3] [eq. 5]
    return 0.3 * (1 - ssim(clean, refined)) + 0.7 * l1(clean, refined)

Hyperparameters

Results

Author-reported. Setup: synthetic test set of 1,525 pairs from the Cityscapes test split, all baselines retrained on the authors' synthetic training set, PSNR and SSIM §5.1 tab. 1. Real images: the dataset of Qian et al., with the authors' models trained on low-accuracy masks §5.2 tab. 2.

Synthetic dataset tab. 1:

Detection on the synthetic set: average precision 0.9973 for the mask §5.1 fig. 6. Efficiency on one 256×512 image, mean of 20 runs on the TITAN Xp: ours 69.74 ms; Pix2Pix 79.1 ms; Qian et al. 97.16 ms; Eigen et al. 107.712 ms; DID-MDN 133.64 ms §5.1, Efficiency. The paper does not say which configuration was timed.

Real images, dataset of Qian et al. tab. 2:

The authors call this "on par" with Qian et al. and note that the dataset is not very challenging, since its training and test sets are very similar §5.2.

Paper vs code

No official code.

Searched: arXiv metadata has no code link; the LaTeX source has no URLs except bibliography entries; GitHub repository search for "raindrop removal", "adherent raindrop", "raindrop synthetic dataset" and "photorealistic raindrop", plus a user search for the first author. The one user match has no repositories and no affiliation shown, so it was not used. Several third-party raindrop repositories came up (e.g. attentive-GAN reimplementations and a synthetic raindrop generator); none is attributed to the authors, so none is used.

Pitfalls

• Dataset generation is under-specified. The paper gives parameter ranges but not the sampling distribution, whether ψ, τ and r are drawn per drop or per image, how many images or drop layouts are generated per background, or the refraction and total-reflection equations. No renderer or dataset release is mentioned. An implementer must rebuild the ray tracer from the prose §3.
• Drop size unit. §1 gives drop sizes of 0.8 to 1.5 centimetres; §3 gives r in [0.8, 1.5] with no unit, and whether r is radius or diameter is not clear. Treat the unit as not stated.
• Steps per epoch do not reconcile. The paper says 50 epochs of 50K steps each §4.4, but also about 30,000 training images at batch size 32 §3. One pass over the training images is about 940 steps. The number of samples per epoch is not stated.
• Learning rate after epoch 40. The schedule is stated only up to epoch 40 §4.4. Whether it stays at 1e-4 for epochs 40 to 50 is not stated.
• Which mask weights the reconstruction loss? Eq. 3 weights pixels that "belong to a raindrop" §4.2 without naming the mask. Using the ground-truth mask is the most likely reading, but the detector's output is possible too. Test-time behaviour is not stated.
• Binarisation of the detector output is not stated. The detector produces a probability mask §4.1, but the dilation and blend steps §4.3 use a binary M̂. No threshold is given.
• Dilation kernel is not stated. It is called a "basic" morphological operation §4.3. Shape and size are unknown; the effect is shown only in fig. 5. Its use at test time is implied by it being a layer in the network but is not stated explicitly.
• Symbol overload. §4 defines R as the raindrop layer in O = (1 − M)B + R §4. From Eq. 2 on, R is the raindrop image. Implement Eq. 4 and the dataset formula as written, using R = raindrop image.
• Notation in Eq. 5. The term 1 − L_ssim treats L_ssim as a similarity, yet the paper calls it a loss. SSIM window size, constants and per-channel use are not stated. The α = 0.3 weight multiplies the SSIM term §4.3.
• Reconstruction loss normalisation. Eq. 3 averages over n pixels; whether channels are summed or averaged is not stated.
• Edge map borders. Eq. 2 uses neighbours at x ± 1 and y ± 1. How the image border is handled is not stated.
• Crop vs full image at test time. Training uses 100×100 crops §4.4, but the efficiency test uses 256×512 §5.1. Which input size produced the Table 1 and Table 2 numbers is not stated.
• Two-stage ordering. The paper says stage 1 trains the detector and reconstruction nets "respectively" §4.4. Whether they train at the same time or one after the other is not stated.
• Undefined network details. Kernel sizes, channel widths of the first convolutions, output activations, and whether the refine network sees the mask as well as the blend are not stated, apart from the layer counts above.
• Ablation text overstates the table. The text says both PSNR and SSIM rise as modules are added §5.1, Ablation. In Table 1 the SSIM falls slightly from 0.9852 (3RN only) to 0.9848 (3RN + RDN + RFN) tab. 1. The claim of a 9.4% PSNR gain for the full model over 3RN only is consistent with the table (41.29 / 37.73).
• Overclaim in the introduction. The contribution list claims a clear lead over every existing method on all existing datasets and real rainy images §1. On Qian et al.'s dataset, Qian et al. has the higher PSNR (30.82 vs 30.17 for the rough-mask model) tab. 2.
• Real-data mask is approximate. For real images the mask is obtained by subtracting the raindrop image from the ground truth, so it is low accuracy §5.2. The binarisation threshold and any handling of noise are not stated.
• Baselines. Eigen et al., Qian et al., DID-MDN and Pix2Pix are retrained on the authors' data §5.1, but their training schedules and hyperparameters are not stated. Table comparisons therefore depend on unknown reproduction choices.
• Efficiency figures. The paper does not say whether the timing includes all three sub-networks or which variant was timed §5.1.

Lineage

• Builds on: the raindrop imagery model of Roser et al. (2010) and You et al. (2016), used for the physics of the drop model §3. Ray-traced synthesis follows Halimeh et al. (2009) §3. The detector follows I-CNN (Fan et al., 2017), with residual blocks from He et al. (2016) §4.1. The refine loss mixes SSIM and L1 following Zhao et al. (2017) §4.3. None of these papers has a card here.
• Compared against: Eigen et al. (2013), Qian et al. (2018), DID-MDN (Zhang et al., 2018) and Pix2Pix (Isola et al., 2017) §5. None has a card here.
• Superseded by: not known from this wiki.