Diving Deeper into Underwater Image Enhancement: A Survey

In plain words

Light and water do not mix well. Underwater photos come out murky, greenish or bluish, and low in contrast, because water absorbs colours and scatters light. This survey looks at the neural networks that try to clean these photos up. It explains the simple physical models of how a dim, coloured underwater picture is formed, and then goes through the networks people have built, sorted by how they are put together. Some copy the image through an encoder and decoder, some stack repeated blocks, some predict the distance to the camera first, and some use a pair of generative adversarial networks to swap between "underwater" and "clear" looks.

The authors also do something rare for this field: they run the published networks on the same set of real underwater photos and score them with six metrics. The scores disagree with each other, and with what the pictures look like, so the survey argues that the field lacks good datasets and good measures. It ends with a list of open problems.

TL;DR

A survey of 14 deep networks for underwater image enhancement, with a benchmark on the UIEBD dataset using six metrics (PSNR, MSE, SSIM, PCQI, UCIQE, UIQM). No single network leads on all metrics. The authors argue that most networks are trained on synthetic data and that current metrics and datasets are not good enough. Use it as a map of the field and of its open problems, not as a ranking that holds on other data.

Problem

Underwater photos suffer from colour cast, low contrast and backscatter, which get worse with distance §1. Classic physics-based and histogram methods exist, and this survey asks how far deep networks have got. Its answer is that most of them are built on simplified physical models, trained on synthetic data, and judged with metrics that were not designed for underwater images §6.

Method

This is a survey, so there is no single algorithm. The card therefore covers: the image-formation models that the networks rely on §2; the taxonomy of the networks §3; the datasets and metrics §4.1, §4.2; and the benchmark §4.3. A reimplementer will mostly need the formulas and the per-method training settings.

Image-formation models

1. Atmospheric scattering model. The observed image is the scene radiance scaled by a transmission term, plus a global light term weighted by one minus the transmission §2.1 eq. 1. In the homogeneous case, the transmission is an exponential of the attenuation coefficient times the distance §2.1 eq. 2.
2. Simplified underwater model. The same form, but applied per colour channel. The transmission is ten raised to the negative wavelength-dependent attenuation times the distance, so it uses base 10, not base e §2.2 eq. 3 eq. 4.
3. Revised underwater model. The direct signal and the backscatter get separate attenuation coefficients, and the backscatter term depends on the veiling light §2.3 eq. 5. The survey credits this model to Akkaynak and Treibitz §2.3.

# Synthesise a degraded underwater image from a clear image  [eq. 3] [eq. 4]
# Input:  clear (H, W, 3) in [0, 1]; depth (H, W) distance d(x)
# Input:  beta (3,) per-channel attenuation; background (3,) light B
# Output: degraded (H, W, 3)

def synthesise_underwater(clear, depth, beta, background):
    # Per-channel transmission, base 10  [eq. 4]
    transmission = 10 ** (-beta * depth[..., None])   # (H, W, 3)
    # Blend the clear scene with background light  [eq. 3]
    degraded = clear * transmission + background * (1 - transmission)
    return degraded

Network taxonomy

Networks are grouped by architecture rather than by CNN or GAN, because the authors judge that split too coarse §3. Groups: encoder-decoder §3.1, modular or block designs §3.2, multi-branch §3.3, depth- or transmission-guided §3.4, and dual-generator GANs §3.5. Table 1 lists the structural settings of each network tab. 1. The per-network training settings are given in the Hyperparameters section below.

Datasets and metrics

Real-world datasets reviewed: Fish4Knowledge, ULFID, MARIS, Haze-line and UIEBD §4.1. UIEBD has 950 images, of which 890 have reference images picked from 12 enhancement methods by 50 volunteers using majority voting; 60 images are kept as challenging data §4.1. The benchmark uses UIEBD only, because it is the only one of these datasets with references §4.3.

The six metrics are: MSE and PSNR eq. 6 eq. 7; SSIM eq. 8; PCQI eq. 9; UCIQE eq. 10, a no-reference chroma-contrast-saturation score; and UIQM [eq. 11], a no-reference score built from colourfulness, sharpness and contrast §4.2.

# PSNR from MSE on 8-bit images  [eq. 6] [eq. 7]
# Input:  enhanced, reference: arrays of shape (H, W, C), values 0..255
# Output: scalar PSNR in dB

def psnr(enhanced, reference):
    # Sum of squared differences over the N pixels  [eq. 6]
    squared = (enhanced - reference) ** 2
    num_pixels = enhanced.shape[0] * enhanced.shape[1]
    mse = squared.sum() / num_pixels
    # Peak intensity L = 255 for 8-bit images  [eq. 7]
    return 10 * log10(255 ** 2 / mse)

Hyperparameters

Values are as printed in the paper. "not stated" means the paper gives no value. Table 1 tab. 1 gives structure for all 14 networks and is reproduced in full below.

Table 1 tab. 1, reproduced as printed in the paper. "—" marks an item the table does not list, which is not the same as "not stated". ✓ means present.

Loss symbols: ℓgan adversarial, ℓc consistency, ℓW Wasserstein, ℓnui nuisance, ℓr relativistic, ℓg gradient tab. 1 caption.

Results

Author-reported. Benchmark: all methods scored on UIEBD with its reference images §4.3 tab. 2. The paper does not state the image subset or split used for the scores. Only methods with available source code or executables, or with author agreement, were included §4.3, footnote 8. Rows marked UWCNN are separate models trained on different synthetic water types.

Table 2 tab. 2, reproduced as printed. Higher is better for PSNR, SSIM, PCQI, UCIQE and UIQM; lower is better for MSE.

Best and second-best per column, as marked in the paper tab. 2:

• PSNR: DUIENet best (19.29); MCycleGAN second (18.33).
• MSE: DUIENet best (1012.20); MCycleGAN second (1132.21).
• SSIM: DUIENet best (0.8093); Original second (0.6168).
• PCQI: Original best (1.1118); URCNN second (1.0936).
• UCIQE: DenseGAN best (0.6291); UWGAN second (0.5921).
• UIQM: URCNN best (1.5332); DUIENet second (1.2963).

The paper's own summary is that DUIENet is best among the competitors §4.3. That holds for the three full-reference scores only; see Pitfalls.

Qualitative results (no references for Haze-line and ULFID) are given as Figures 4 to 8 fig. 4 fig. 5 fig. 6 fig. 7 fig. 8. In words: GAN outputs shift tone more than CNN outputs, which the authors link to higher PSNR for CNNs §4.4; bluish images remain bluish, with DUIENet and UWCNN best but still bluish at distance §4.4; no method cleans high backscatter §4.4; all deep methods fall behind the conventional Haze-line method on that dataset §4.4; and on ULFID, all methods remove green cast but amplify noise §4.4.

Paper vs code

No official code. The authors' repository saeed-anwar/UWSurvey (cloned at commit 181c5c7b5046271cb0521abab1830c4109ef21e0) holds only a README and figure images. It has no source code and no LICENSE file, so no code field is set. The README cites the paper and asks readers to cite it, but contains no training, evaluation or benchmark scripts. The benchmark numbers in Table 2 therefore cannot be reproduced from released code.

Searched: the paper and its source for code links (the source's commented-out link points to a different survey, SRsurvey, and was not used); the authors' GitHub repository lists (saeed-anwar, li-chongyi), instead of the GitHub search API, which the search tool could not reach; the repository above. The other underwater repositories on those pages (UWCNN, Water-Net_Code, JOE2021_ACDC) belong to separate papers, so they are not cited for this survey.

Pitfalls

• Survey, not method. The benchmark numbers come from the authors running each model. Only methods with code or executables, or author agreement, were included §4.3, footnote 8. Several entries are not public models, so this is not a full comparison §3.6, footnote 2.
• Benchmark evaluation split. The subset and split used for Table 2 are not stated §4.3. Reference images come from 12 enhancement methods, chosen by volunteers §4.1. This is an inference, not a claim in the paper: reference scores may favour methods like those 12.
• Output scale. The authors resize outputs that are scaled-down relative to the input, but the scale and the resize method are not stated §4.3.
• Table 2 and the text disagree. The text calls DUIENet best among the competitors §4.3. Table 2 shows Original best on PCQI, DenseGAN best on UCIQE and URCNN best on UIQM tab. 2. The authors also say these metrics disagree with the visual results §5.
• Identical UCIQE values. Original, MCycleGAN and URCNN all show 0.5196 for UCIQE tab. 2. The paper does not explain this; it may be a copy error.
• "Original" row. The row is not defined in the text. It is presumably the unprocessed input, but the paper does not say so.
• DenseGAN loss. Text lists GAN, ℓ1 and gradient losses §3.2. Table 1 lists ℓ2, GAN and gradient tab. 1. Check the original code before use; the code was not available for this survey.
• DUIENet depth. Text says 14 convolutional layers §3.3. Table 1 says depth 8 tab. 1. The input size is printed as "112×11" in the text, while Table 1 says 112×112 §3.3 tab. 1.
• P2P data counts. The text says 3,359 images were collected, then 10,000 training and 2,000 test images were selected from them §3.1. That is more than were collected. The counts are not reconcilable from the paper.
• UWCNN variant count. Text says ten models, one per water type §3.2. Table 2 lists eight UWCNN variants (types 1, 3, 5, 7, 9, I, II, III) tab. 2.
• Learning-rate schedules. UIE-Net's schedule reads "5×0³" in the text §3.3; the intended value is not clear. URCNN's schedule is printed as "reduced to 10⁻⁴ for 60 epochs" §3.4; the sequence is ambiguous.
• Base e versus base 10. The homogeneous model uses e to the power of minus beta times distance eq. 2. The simplified model uses 10 to the power of minus the wavelength-dependent coefficient eq. 4. Do not mix them: the coefficients have different meanings.
• Equation numbers. Numbers 1 to 11 assume sequential numbering across the paper, which is the usual default for this class. The compiled PDF was not in the source, so this is not checked against a rendered copy. Figure and table numbers were counted from the source in order.
• SSIM and UCIQE/UIQM constants. The SSIM constants C1 to C3 eq. 8, the UCIQE weights C1 to C3 eq. 10 and the UIQM weights c1 to c3 eq. 11 are not stated. The UIQM weights are called "application dependent" §4.2.
• Chroma wording. The text says human perception tracks the variance of chroma §4.2, but UCIQE eq. 10 uses the standard deviation. Follow the equation.
• PCQI labels. The text lists mean, signal strength and structure as the three quantities §4.2, then assigns the components q_c to "structural distortion" and q_s to "contrast change" eq. 9. The mapping is ambiguous. Check the original PCQI paper before implementing.
• Metric channel handling. For MSE and PSNR, the paper does not say whether N counts pixels only or pixel-channels eq. 6.
• Missing training details. Batch sizes, epochs and optimiser settings are often "not stated" tab. 1 §3. Several losses have no weights stated.
• Depth and parameter comparisons. The survey notes networks are shallow, under 45 layers, with WaterGAN deepest at 42 §3.6. Parameter counts cannot be compared, because most models are not public §3.6, footnote 2.
• Synthetic training. Most listed networks train on synthetic data, which the authors link to limited generalisation §5, §6.

Lineage

• Builds on: the atmospheric scattering model (Koschmieder 1924, cited §2.1), the simplified underwater model (a prior work, cited §2.2) and the revised model (Akkaynak and Treibitz, cited §2.3). None of these are in this wiki.
• Surveyed networks: UIE-Net, UIR-Net, P2P, UIE-sGAN, WaterGAN, UGAN, UWCNN, URCNN, UWGAN, DUIENet, MCycleGAN, DenseGAN, FGAN, UIE-DAL §3 tab. 1.
• Later underwater cards in this wiki, not compared in this survey: /papers/2111.11843 (U-shape Transformer), /papers/2103.07138 (UIEC²-Net), /papers/2208.06857 (URanker).
• Superseded by: not known from content in this wiki.