Deep image prior

Overview

The deep image prior (DIP) idea is to use a neural network's structure as the prior, rather than a trained model. The network starts random and is fitted to one observation. Its weights are the variables, and the input is a fixed random code that is never optimised 1806.06438 §3.1. The authors argue that convolutional generators are biased toward smooth, natural-looking images, so the architecture alone favours plausible reconstructions [arxiv:1806.06438 §3.1]. The original untrained-network idea is due to Ulyanov et al. (2017), linked on arXiv: arXiv:1711.10925. This page takes its content only from the compressed-sensing card below.

The shared core of the variants is a data-fit term for a known linear measurement y = Ax + noise, a penalty on the output, and a fixed step budget. Stopping early is part of the method, not an afterthought: the paper stops at a fixed number of steps and does not use a validation rule [arxiv:1806.06438 §3.1] app. A. Only one card lists this slug so far, so every variant below comes from the same paper.

Variants

Unlearned fit (λ_L = 0)

The generator is a DCGAN-style network with random weights and a fixed random latent input. Its weights are fitted to the measurements, with total variation weighted by λ_T, and nothing is trained beforehand 1806.06438 §3.1 eq. 2. The paper's settings are RMSProp with learning rate 1e-3 and momentum 0.9, 1000 steps, and λ_T = 0.01 1806.06438 §5.1 app. A. Against unlearned baselines, the authors report that it wins on MNIST in almost all cases, and on chest X-rays it wins against all baselines except BM3D-AMP at higher sampling rates 1806.06438 §5.2.2. Card: Compressed Sensing with Deep Image Prior and Learned Regularization.

Learned regularisation (λ_L > 0)

The second variant adds a quadratic penalty, weighted by λ_L, that pulls the weights toward a mean μ with covariance Σ. This is the MAP estimate under a Gaussian prior on the weights, and with μ = 0 and Σ = I it reduces to ℓ2 weight decay 1806.06438 §3.2 eq. 4. The prior is estimated from the measurements of about ten images, each fitted with the unlearned method, with no ground truth used 1806.06438 §3.2.1 1806.06438 §3.2.2. The paper uses λ_L = 100 for the X-ray experiments 1806.06438 §5.2.1 [app. A]. Author-reported error reductions from adding this term, on X-rays with Gaussian measurements, range from 0.2% to 37.4%. The largest gains are at high noise and few measurements, which the authors name as the regime where it helps most 1806.06438 tab. 1 1806.06438 §5.2.1. The number of samples S and rounds T in the prior estimate are not stated [arxiv:1806.06438 alg. 1].

When to use which

Runtime numbers from the paper: at m = 1000, CS-DIP takes 15.6 s against 13.8 s for TVAL3 and 51.1 s for BM3D-AMP; at m = 8000, CS-DIP takes 29.9 s against 56.7 s for TVAL3 1806.06438 tab. 2.

Implementation sketch

The loop shared by the variants. The learned term is an optional addition to the loss. The step budget and learning rate are the choices that the paper reports and that the card's Pitfalls section warns about.

# Shared core of the deep image prior  [arxiv:1806.06438 §3.1] [eq. 2]
# Input:  y (m,), A (m, n) known and differentiable, step budget
# Output: image estimate x (H, W), values in [-1, 1]

def deep_image_prior(y, A, steps, lambda_T, prior=None, lambda_L=0):
    # Fixed random input; only the weights are optimised  [§3.1]
    z = random_latent(z_dim)
    G = untrained_generator()              # random initial weights  [§5.1]
    opt = RMSProp(G.weights, lr=1e-3, momentum=0.9)  # [§5.1]
    for _ in range(steps):                 # fixed budget, not validation  [app. A]
        x = G(z)
        loss = sum((y - A @ flatten(x)) ** 2)  # data fit  [eq. 1]
        loss += lambda_T * total_variation(x)  # TV penalty  [eq. 2]
        if prior is not None:
            # Quadratic penalty toward (mu, Sigma) on the weights  [eq. 4]
            loss += lambda_L * quad_penalty(G.weights, prior)
        opt.step(loss)
    return G(z)

Contradictions

None known across papers, since only one card lists this slug. Inside the card, the paper and its released code differ on several settings: the latent size, the TV and learned-regularisation weights, the number of random restarts, and the stopping rule. The card lists each one, and this page does not choose between them 1806.06438 Paper vs code. The paper's MNIST results are in the unlearned section, but the code runs MNIST with λ_L = 0 [arxiv:1806.06438 §5.2.2] 1806.06438 Paper vs code.

Papers

• Compressed Sensing with Deep Image Prior and Learned Regularization: applies an untrained DCGAN-style generator to linear measurements, with an optional learned Gaussian prior on the weights.