---
kind: concept
slug: generative-prior-inverse-problems
title: Generative priors for inverse problems
summary: >-
  A generative prior is reused at inference, and a measurement or constraint enters as a gradient term. Three forms: posterior sampling by annealed Langevin dynamics for a known linear model; a point estimate under a differentiable constraint; and a diffusion-generated target fitted by a differentiable renderer, a looser match. Covers when to use which and where code departs from the paper.
tags: [inverse-problems, diffusion, posterior-sampling, plug-and-play, differentiable-rendering]
papers: ["2108.01368", "2206.09012", "2404.17569"]
related: [diffusion-models, classifier-guidance, classifier-free-guidance, constraint-satisfaction, differentiable-rendering, image-denoising, medical-image-synthesis, promptable-segmentation, semantic-segmentation]
status: draft
updated_at: 2026-10-19
---

## Overview

An inverse problem asks for a signal x that explains some observation y, for
example an image from partial MRI measurements or a labelling that satisfies
weak labels. A generative model supplies the prior, which says what plausible
x look like. In the first two forms the observation enters as a second gradient
term, so the generator is never retrained for the new task
[arxiv:2206.09012 §1] [arxiv:2108.01368 §2.2].

The shared invariant: a gradient of the data term with respect to the current
estimate drives the fit. The first two forms also query the prior at a sequence
of noise levels that falls from high to low. The first uses a known linear
forward model [arxiv:2108.01368 §2.1] [eq. 1]. The second requires a
differentiable constraint [arxiv:2206.09012 §2.2]. The third form fits a
structured output, a material graph, to an image that a diffusion model made
once. It has no noise-level loop, so it matches the invariant only loosely
[arxiv:2404.17569 §3.3].

The three cards in this wiki apply the pattern to MRI reconstruction, to
conditional digits, face attributes, land-cover segmentation, a tour problem,
and procedural materials for 3D meshes. They solve different outputs, so their
reported numbers are not comparable.

## Variants

### Posterior sampling with annealed Langevin dynamics

Used when the observation is a linear measurement with noise. The sampler
draws from the posterior, not from a single best estimate. It runs annealed
Langevin steps on a score-based prior, adding a data-consistency term and
injected noise at each level [arxiv:2108.01368 §2.2] [eq. 4]. The prior is a
score network trained on T2-weighted brain slices, and the same weights serve
other sampling patterns and anatomies [arxiv:2108.01368 §4] [arxiv:2108.01368
§4.2]. Repeated runs from different starting noise give a spread of images,
which the card uses as a per-pixel uncertainty estimate [arxiv:2108.01368
§4.3]. The reported cost is 16 minutes and 0.95 GB for one high-resolution
brain on an RTX 2080Ti [arxiv:2108.01368 §4.1]. Card:
[Robust Compressed Sensing MRI with Deep Generative Priors](/papers/2108.01368).

### Point estimate under a differentiable constraint

Used when the target is a single output and the constraint is any
differentiable function of the sample. A pretrained denoising diffusion model
gives the prior term, which is the summed denoising error over noise levels.
The constraint enters as a log term, and the estimate is optimised by
gradient steps with the noise level annealed from high to low
[arxiv:2206.09012 §2.2] [eq. 12] [alg. 1]. The card instantiates the
constraint in four ways: hand-crafted symmetry terms and a classifier
likelihood for conditional digits [arxiv:2206.09012 §3.1], classifier
log-likelihoods for face attributes [arxiv:2206.09012 §3.2] [eq. 14], a
mutual-information and weak-label term for land-cover segmentation
[arxiv:2206.09012 §4] [eq. 15], and a soft tour-matrix construction for the
travelling salesman problem [arxiv:2206.09012 §5]. Card:
[Diffusion models as plug-and-play priors](/papers/2206.09012).

For a different way to steer a diffusion sampler with a classifier, see
[classifier guidance](/concepts/classifier-guidance).

### Fit to a generated target by differentiable rendering

Used when the output is a structured object that a renderer can draw, and the
generative model is needed only to produce a target image. A diffusion model
steered by a segment mask paints one view of a 3D mesh [arxiv:2404.17569 §3.1]
[eq. 1]. The painted image then becomes the target for a fit. The output is a
procedural material graph whose textures are rendered under a learned light
field and compared with the target, using an albedo term and a masked L1 plus
style term [arxiv:2404.17569 §3.3] [eq. 2]. Unlike the two forms above, the
diffusion model runs once, outside any optimisation loop, and the gradient
comes from the renderer, not from the prior. The graphs stay editable in
modelling software [arxiv:2404.17569 §3.5]. The authors report 7 minutes on
average per shape [arxiv:2404.17569 §4.4]. Card:
[MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](/papers/2404.17569).

## When to use which

| Situation | Use | Why |
|---|---|---|
| The observation is a known linear operator with noise, and you need a spread of plausible outputs | Posterior sampling | The card draws several samples and reads uncertainty from their spread [arxiv:2108.01368 §4.3] |
| You need one output per run and the constraint is not a measurement model | Point estimate | The constraint is a log term on the sample, which the card uses for attributes, segmentation and tours [arxiv:2206.09012 §2.2] [arxiv:2206.09012 §3.2] [arxiv:2206.09012 §5] |
| The output is a structured, editable object that a renderer can draw, and the target is an image | Generated target, fitted by rendering | The graph is fitted to a diffusion-painted view, and it stays editable [arxiv:2404.17569 §3.3] [arxiv:2404.17569 §3.5] |
| One prior must serve several sampling patterns or anatomies | Posterior sampling | The score model is trained without the sampling pattern and reused across tests [arxiv:2108.01368 §4.2] [arxiv:2108.01368 §6] |
| Little or no domain data for the generator | Neither as shown | The MRI prior used 14,539 brain slices [arxiv:2108.01368 §4]. The point-estimate card's from-scratch land-cover run uses 640 samples and reaches 76.0% accuracy on Durham, against 79.8% when fine-tuned from Pittsburgh labels [arxiv:2206.09012 app. B.3] [tab. 1]; the face-attribute case used a pretrained FFHQ denoiser [arxiv:2206.09012 §3.2] |
| Latency budget is seconds, not minutes | None as reported | The brain reconstruction takes minutes [arxiv:2108.01368 §4.1]; the point-estimate card reports 200 to 1000 denoiser steps per sample [arxiv:2206.09012 app. B.1] [arxiv:2206.09012 app. B.2]; the material card reports 7 minutes per shape [arxiv:2404.17569 §4.4] |
| The constraint is not differentiable | None | All three forms need gradients, of the data term, the constraint or the renderer [arxiv:2108.01368 §2.2] [arxiv:2206.09012 §2.2] [arxiv:2404.17569 §3.3] |

## Implementation sketch

The common core is a loop over decreasing noise levels with two gradient
terms, one from the prior and one from the observation or constraint. The
variants differ in whether noise is injected and whether the output is a
sample or a single iterate. The third form does not fit this loop: it has no
noise schedule, and its loop is ordinary optimisation of a renderer loss
against a fixed target.

```pseudocode
# Common core of annealed inverse-problem samplers  [eq. 4] [alg. 1]
# Input:  prior score or denoiser, observation y, data term, schedule
# Output: one reconstruction x

def reconstruct(prior, y, data_grad, schedule, lr):
    # Start from noise: a posterior draw or a point-estimate start  [§2.2]
    x = random_normal(shape)
    for level in schedule:
        # Prior term at the current noise level  [eq. 4] [eq. 12]
        prior_term = prior(x, level)
        # Gradient of the observation or constraint  [eq. 4] [eq. 15]
        data_term = data_grad(x, y)
        # Posterior sampling adds noise here; point estimates do not  [alg. 1]
        x = x + lr * (prior_term + data_term) + noise_for(lr)
    return x
```

Notes on the variants:

- **Posterior sampling.** The data term is the adjoint of the forward model
  applied to the residual, divided by sigma squared plus gamma_t squared
  [arxiv:2108.01368 §2.2] [eq. 4]. The paper leaves sigma and gamma_t without
  values, so they must be set by the implementer.
- **Point estimate.** The prior term is the denoising error, which is
  summed over noise levels and redrawn at each step [arxiv:2206.09012 §2.2].
  The stochastic draw means repeated runs can return different estimates.
- **Generated target.** The diffusion model is called once, before the fit, to
  produce the image that the graph is matched to. The fit then minimises a
  renderer loss against that fixed image, with an albedo term and a masked L1
  plus style term [arxiv:2404.17569 §3.3].
- **Schedules.** The card for the point estimate states that the paper gives
  no time-schedule formula for the MNIST setup and that the TSP schedule is
  resampled from it [arxiv:2206.09012 app. B.1] [arxiv:2206.09012 app. B.4].
  Take the schedule from the released code and record it. The generated-target
  form has no schedule.

## Contradictions

Not between the cards' claims. They target different outputs, so their results
are not directly comparable. The released code of the first card replaces the
stated data-term scale with a normalised rule and a fixed multiplier. The card
states that an implementer who follows only the paper's equation will not get
the published numbers [arxiv:2108.01368 §2.2] [eq. 4]. The second card's
released loss uses a mean where the paper uses a sum, which changes the effect
of the constraint weights [arxiv:2206.09012 eq. 10] [arxiv:2206.09012 eq. 12].
Both are paper-versus-code gaps, not disagreements between the papers.

The third form's FID and KID compare rendered images with "the real image", but
the paper does not name the reference set [arxiv:2404.17569 §4.2]. Its numbers
cannot be set beside the other two cards' results.

No official code was found for the third form, so its description follows the
paper alone.

## Papers

- [Robust Compressed Sensing MRI with Deep Generative Priors](/papers/2108.01368): posterior sampling with a score-based prior for MRI reconstruction, reported as robust under sampling and anatomy shifts.
- [Diffusion models as plug-and-play priors](/papers/2206.09012): a pretrained denoising diffusion model as a prior, combined with differentiable constraints at inference.
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](/papers/2404.17569): a diffusion model paints a mesh view, and a procedural material graph is retrieved and fitted to it by differentiable rendering.