Generative priors for inverse problems

Overview

An inverse problem asks for a signal x that explains some observation y, for example an image from partial MRI measurements or a labelling that satisfies weak labels. A generative model supplies the prior, which says what plausible x look like. In the first two forms the observation enters as a second gradient term, so the generator is never retrained for the new task 2206.09012 §1 2108.01368 §2.2.

The shared invariant: a gradient of the data term with respect to the current estimate drives the fit. The first two forms also query the prior at a sequence of noise levels that falls from high to low. The first uses a known linear forward model 2108.01368 §2.1 eq. 1. The second requires a differentiable constraint 2206.09012 §2.2. The third form fits a structured output, a material graph, to an image that a diffusion model made once. It has no noise-level loop, so it matches the invariant only loosely 2404.17569 §3.3.

The three cards in this wiki apply the pattern to MRI reconstruction, to conditional digits, face attributes, land-cover segmentation, a tour problem, and procedural materials for 3D meshes. They solve different outputs, so their reported numbers are not comparable.

Variants

Posterior sampling with annealed Langevin dynamics

Used when the observation is a linear measurement with noise. The sampler draws from the posterior, not from a single best estimate. It runs annealed Langevin steps on a score-based prior, adding a data-consistency term and injected noise at each level 2108.01368 §2.2 eq. 4. The prior is a score network trained on T2-weighted brain slices, and the same weights serve other sampling patterns and anatomies 2108.01368 §4 [arxiv:2108.01368 §4.2]. Repeated runs from different starting noise give a spread of images, which the card uses as a per-pixel uncertainty estimate [arxiv:2108.01368 §4.3]. The reported cost is 16 minutes and 0.95 GB for one high-resolution brain on an RTX 2080Ti 2108.01368 §4.1. Card: Robust Compressed Sensing MRI with Deep Generative Priors.

Point estimate under a differentiable constraint

Used when the target is a single output and the constraint is any differentiable function of the sample. A pretrained denoising diffusion model gives the prior term, which is the summed denoising error over noise levels. The constraint enters as a log term, and the estimate is optimised by gradient steps with the noise level annealed from high to low 2206.09012 §2.2 eq. 12 alg. 1. The card instantiates the constraint in four ways: hand-crafted symmetry terms and a classifier likelihood for conditional digits 2206.09012 §3.1, classifier log-likelihoods for face attributes 2206.09012 §3.2 eq. 14, a mutual-information and weak-label term for land-cover segmentation 2206.09012 §4 eq. 15, and a soft tour-matrix construction for the travelling salesman problem 2206.09012 §5. Card: Diffusion models as plug-and-play priors.

For a different way to steer a diffusion sampler with a classifier, see classifier guidance.

Fit to a generated target by differentiable rendering

Used when the output is a structured object that a renderer can draw, and the generative model is needed only to produce a target image. A diffusion model steered by a segment mask paints one view of a 3D mesh 2404.17569 §3.1 eq. 1. The painted image then becomes the target for a fit. The output is a procedural material graph whose textures are rendered under a learned light field and compared with the target, using an albedo term and a masked L1 plus style term 2404.17569 §3.3 eq. 2. Unlike the two forms above, the diffusion model runs once, outside any optimisation loop, and the gradient comes from the renderer, not from the prior. The graphs stay editable in modelling software 2404.17569 §3.5. The authors report 7 minutes on average per shape 2404.17569 §4.4. Card: MaPa: Text-driven Photorealistic Material Painting for 3D Shapes.

When to use which

Implementation sketch

The common core is a loop over decreasing noise levels with two gradient terms, one from the prior and one from the observation or constraint. The variants differ in whether noise is injected and whether the output is a sample or a single iterate. The third form does not fit this loop: it has no noise schedule, and its loop is ordinary optimisation of a renderer loss against a fixed target.

# Common core of annealed inverse-problem samplers  [eq. 4] [alg. 1]
# Input:  prior score or denoiser, observation y, data term, schedule
# Output: one reconstruction x

def reconstruct(prior, y, data_grad, schedule, lr):
    # Start from noise: a posterior draw or a point-estimate start  [§2.2]
    x = random_normal(shape)
    for level in schedule:
        # Prior term at the current noise level  [eq. 4] [eq. 12]
        prior_term = prior(x, level)
        # Gradient of the observation or constraint  [eq. 4] [eq. 15]
        data_term = data_grad(x, y)
        # Posterior sampling adds noise here; point estimates do not  [alg. 1]
        x = x + lr * (prior_term + data_term) + noise_for(lr)
    return x

Notes on the variants:

• Posterior sampling. The data term is the adjoint of the forward model applied to the residual, divided by sigma squared plus gamma_t squared 2108.01368 §2.2 eq. 4. The paper leaves sigma and gamma_t without values, so they must be set by the implementer.
• Point estimate. The prior term is the denoising error, which is summed over noise levels and redrawn at each step 2206.09012 §2.2. The stochastic draw means repeated runs can return different estimates.
• Generated target. The diffusion model is called once, before the fit, to produce the image that the graph is matched to. The fit then minimises a renderer loss against that fixed image, with an albedo term and a masked L1 plus style term 2404.17569 §3.3.
• Schedules. The card for the point estimate states that the paper gives no time-schedule formula for the MNIST setup and that the TSP schedule is resampled from it 2206.09012 app. B.1 2206.09012 app. B.4. Take the schedule from the released code and record it. The generated-target form has no schedule.

Contradictions

Not between the cards' claims. They target different outputs, so their results are not directly comparable. The released code of the first card replaces the stated data-term scale with a normalised rule and a fixed multiplier. The card states that an implementer who follows only the paper's equation will not get the published numbers 2108.01368 §2.2 eq. 4. The second card's released loss uses a mean where the paper uses a sum, which changes the effect of the constraint weights 2206.09012 eq. 10 2206.09012 eq. 12. Both are paper-versus-code gaps, not disagreements between the papers.

The third form's FID and KID compare rendered images with "the real image", but the paper does not name the reference set 2404.17569 §4.2. Its numbers cannot be set beside the other two cards' results.

No official code was found for the third form, so its description follows the paper alone.

Papers

• Robust Compressed Sensing MRI with Deep Generative Priors: posterior sampling with a score-based prior for MRI reconstruction, reported as robust under sampling and anatomy shifts.
• Diffusion models as plug-and-play priors: a pretrained denoising diffusion model as a prior, combined with differentiable constraints at inference.
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes: a diffusion model paints a mesh view, and a procedural material graph is retrieved and fitted to it by differentiable rendering.