Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

In plain words

Estimating depth from one photo means guessing how far away each pixel is. Models trained on a few datasets tend to fail on scenes that look unlike their training data. This paper starts from a large image generator, Stable Diffusion, which has learned a broad picture of what the visual world looks like. It retrains that generator to output a depth map instead of a picture, using only computer-generated depth data. The model is called Marigold.

Its depth values are correct up to an unknown scale and offset. That gives the right layout of a scene, but not distances in metres, unless extra information is supplied. The output is random, so the model is run several times and the answers are merged. Merging improves accuracy and costs time.

On photos from five real datasets, Marigold is the most accurate model in most cases, but it is clearly weaker on one of them, DIODE. The numbers come from the authors.

TL;DR

• Marigold fine-tunes only the denoising U-Net of Stable Diffusion v2 to predict affine-invariant depth. Training uses only synthetic Hypersim and Virtual KITTI data §3.3 §4.2.
• The image and the depth map are both encoded with the frozen VAE. Their latents are concatenated at the U-Net input §3.2.
• Inference runs the model 10 times from different noise, then aligns the runs by scale and shift and takes the median. The paper uses 50 DDIM steps per run §3.4 §4.1.
• Author-reported: with the 10-run ensemble it has the best average rank (1.4) and the lowest AbsRel on NYUv2, KITTI, ETH3D and ScanNet. On DIODE it is worse than DPT tab. 1.
• The official code is Apache-2.0, but several settings differ from the paper. Its model weights carry a separate OpenRAIL++-M license, which this card does not cover LICENSE-MODEL.txt.

Problem

Depth from a single image is ill-posed. The camera's geometry and the scene's size are both unknown, so a model has to rely on prior knowledge about shapes, layouts and occlusion §1. Earlier estimators learned this from narrow domains or from mixtures of real RGB-D datasets, and they struggle on unfamiliar content §1.

The authors ask whether the prior in a text-to-image diffusion model can replace that data. Their hypothesis is that a model trained to generate broad image content already encodes the visual knowledge that depth needs §1.

Two design choices follow from the problem. The output is affine-invariant, meaning it is correct up to a global scale and offset, because scale bars are often unavailable for in-the-wild images §1. Training uses synthetic depth, because real sensors leave holes and noise in the ground truth §3.3.

Method

Marigold is a latent diffusion model that is conditioned on the input image. Training follows the standard denoising objective, but the model is trained on depth latents instead of pixels §3.1.

Generative formulation §3.1. A depth map d is noised in the forward process to d_t with a variance schedule over T steps eq. 1. The model predicts the noise from the noisy depth, the image and the timestep eq. 2. Both depth and image are encoded into latents with a VAE, so the denoiser works in latent space §3.1.

Architecture §3.2. The Stable Diffusion VAE is kept frozen. A single-channel depth map is replicated into three channels so it fits the RGB encoder. After denoising, the decoder output is averaged over its three channels to give one depth map §3.2. The U-Net's first convolution is widened from 4 to 8 input channels. The new weights copy the original ones and are halved, so the activations keep their scale §3.2.

Normalisation §3.3 eq. 3. Each ground-truth depth map is mapped linearly so that its 2nd and 98th percentiles fall at -1 and 1. The map is then clipped to that range. The normalisation is per image, which makes the target affine-invariant §3.3.

Synthetic training data §3.3. Only synthetic depth is used, because it is dense and clean. The paper treats the domain gap to real data as the main risk §3.3.

Annealed multi-resolution noise §3.3 app. A.2. The forward noise is the sum of several Gaussian noise images at different resolutions, each upsampled to the U-Net input size. A level i above zero gets the weight (s t / T)^i, where s is a strength below one and t is the timestep. The lowest-resolution noise therefore fades out near the clean end of the schedule. The paper gives no value for s app. A.2.

Inference §3.4. The image is encoded once. The depth latent starts from standard Gaussian noise, not from the multi-resolution noise used in training. The denoiser then runs with respaced DDIM steps §3.4. The final latent is decoded once and averaged over channels §3.4.

Test-time ensembling §3.4 eq. 4. Runs with different initial noise differ, so several runs are merged. Each run gets its own scale and shift. These are chosen to minimise the distance between every pair of aligned runs, plus a penalty that keeps the merged map inside the unit range eq. 4. The merged result is the pixel-wise median of the aligned runs §3.4. No ground truth is needed §3.4.

# Denoising step during training  [§3.1] [§3.3] [eq. 2]
# Input:  rgb (B, 3, H, W); depth (B, H, W) normalised to [-1, 1]  [eq. 3]
# Output: scalar loss

def training_step(rgb, depth):
    # Encode the image and the depth with the frozen VAE  [§3.2]
    rgb_latent = vae_encode(rgb)                  # (B, 4, h, w)
    depth_latent = vae_encode(repeat_3(depth))    # (B, 4, h, w)

    # Draw one timestep per image  [§3.1]
    timesteps = random_int(0, num_steps, batch)   # num_steps = 1000  [§4.1]

    # Annealed multi-resolution noise; s is not stated in the paper  [app. A.2]
    strength = s * timesteps / num_steps
    noise = multi_res_noise(depth_latent, strength)

    # Forward diffusion of the depth latent  [eq. 1]
    noisy_latent = add_noise(depth_latent, noise, timesteps)

    # The U-Net sees the image latent next to the noisy depth latent  [§3.2]
    unet_input = concat(rgb_latent, noisy_latent) # (B, 8, h, w)
    prediction = unet(unet_input, timesteps, empty_text)

    # Regress the target and average the squared error  [§4.1] [eq. 2]
    target = velocity_target(depth_latent, noise, timesteps)
    return mean((prediction - target) ** 2)

# Inference with test-time ensembling  [§3.4] [eq. 4]
# Input:  rgb (1, 3, H, W); runs = 10, steps = 50  [§4.1]
# Output: affine-invariant depth (H, W) in [0, 1]

def predict_depth(rgb, runs=10, steps=50):
    # Encode the image once  [§3.4]
    rgb_latent = vae_encode(rgb)
    depths = []
    for run in range(runs):
        # Each run starts from fresh standard Gaussian noise  [§3.4]
        depth_latent = random_normal_like(rgb_latent)
        for t in ddim_timesteps(steps):
            # Denoise with the image latent as conditioning  [§3.4]
            noise_pred = unet(concat(rgb_latent, depth_latent), t)
            depth_latent = ddim_step(noise_pred, depth_latent, t)
        # Decode once and average the three channels  [§3.2]
        depths.append(mean_channels(vae_decode(depth_latent)))
    # Align the runs and take the median  [§3.4] [eq. 4]
    return ensemble_align_median(stack(depths))

# Joint scale and shift alignment of N runs  [§3.4] [eq. 4]
# Input:  depths (N, H, W)
# Output: merged depth (H, W)

def ensemble_align_median(depths):
    # Start each run's scale and shift from its range  [code: marigold/util/ensemble.py#L91-L97]
    scales, shifts = initial_scale_shift(depths)
    # Minimise pairwise distance plus a range penalty  [eq. 4]
    # The weight lambda is not stated in the paper
    scales, shifts = minimise(pairwise_cost(depths, scales, shifts))
    aligned = depths * scales + shifts
    merged = median_over_runs(aligned)
    # Rescale the merged map to [0, 1]  [code: marigold/util/ensemble.py#L184-L192]
    return (merged - merged.min()) / (merged.max() - merged.min())

Hyperparameters

Paper values are stated in the text or the appendix. "not stated" means the paper does not give a value. Code values come from the official repository at code_commit.

Results

Author-reported. Zero-shot: trained only on synthetic data, evaluated on five real datasets that were not seen in training §4.2 tab. 1. Metrics are AbsRel (lower is better) and δ1 (higher is better), both in percent. Predictions are aligned to the ground truth by least squares before scoring §4.2. Baselines come from Metric3D, except that ScanNet was re-run on the authors' split tab. 1.

Table 1 marks Marigold with a footnote: its pretrained Stable Diffusion backbone saw image-text data tab. 1. The backbone was pretrained on LAION-5B §3.2. So "no real depth" does not mean "no real data" tab. 1 §3.2.

Ablations §4.3. These run on a zero-shot validation set: the official NYUv2 training split (785 images) and an 800-image subset of the KITTI Eigen training split §4.3. The paper does not say whether these runs use the ensemble.

Training noise tab. 2:

Training data tab. 3:

Mixing ratio tab. S3: 95/5 gives KITTI AbsRel 11.1; 90/10 gives 11.3 on KITTI and 5.6 on NYUv2; 50/50 gives 12.8 on KITTI and 6.0 on NYUv2 tab. S3.

Test-time ensembling §4.3 fig. 6: 10 runs reduce NYUv2 AbsRel by about 8%, and 20 runs by about 9.5%. The gain shrinks beyond 10 runs §4.3.

Denoising steps §4.3 fig. 7: accuracy rises with more steps and saturates at about 10. The paper uses 50 steps anyway §4.1.

Prediction variance app. B.3 tab. S2: across 10 runs on NYUv2, the pixel-wise standard deviation is 0.086 with Gaussian training noise, 0.037 with multi-resolution noise and 0.033 with the annealed version.

VAE check app. B.1: reconstructing 800 Hypersim depth maps through the frozen VAE gives a mean absolute error of 0.0095 ± 0.0091 app. B.1.

Paper vs code

Code: prs-eth/Marigold at code_commit. The owner is the authors' lab (PRS, ETH Zürich), per the repository and its model license LICENSE-MODEL.txt.

• Text conditioning. Paper: text conditioning is disabled §4.1. Code encodes an empty prompt and passes it to the U-Net on every step marigold_depth_trainer.py marigold_depth_pipeline.py.
• Training length. Paper: 18K iterations §4.1. Code: max_iter 30000 and a schedule that decays over 25,000 train_marigold_depth.yaml train_marigold_depth.yaml.
• Learning-rate schedule. Paper gives only the rate §4.1. Code adds 100 warmup steps and exponential decay to 1% train_marigold_depth.yaml lr_scheduler.py.
• Training noise scheduler. Paper: DDPM with 1000 steps §4.1. Code also sets zero-SNR rescaling and trailing timestep spacing marigold_depth_trainer.py.
• Prediction target. Eq. 2 is written as noise prediction eq. 2, but §4.1 says v-objective §4.1. Code picks the target from the scheduler's prediction_type marigold_depth_trainer.py. The config names stable-diffusion-2 without a revision train_marigold_depth.yaml.
• Multi-resolution noise. The weight (s t / T)^i matches app. A.2 marigold_depth_trainer.py, with s = 0.9 train_marigold_depth.yaml. Code adds a random scale factor between 2 and 4 per level, and its downsampling compounds across levels, which the paper does not describe multi_res_noise.py.
• Ensemble cost. Eq. 4 takes the square root of the mean squared pairwise distance eq. 4. Code sums the root-mean-square distance over pairs ensemble.py.
• Ensemble penalty. The paper does not give lambda eq. 4. Code uses 0.02 ensemble.py ensemble.py.
• Output range. The paper relies on the penalty to fix the unit scale §3.4. Code also min-max rescales the median to [0, 1] ensemble.py.
• Inference steps. Paper: 50 DDIM steps §4.1. Code pipeline default: 4 marigold_depth_pipeline.py. The current eval script uses 1 step 11_infer_nyu.sh. An older eval script used 50 steps and 10 runs 11_infer_nyu.sh.
• Checkpoint. The default checkpoint is prs-eth/marigold-depth-v1-1, a newer-named release infer.py. The code does not say which weights reproduce Table 1.
• Latent encoding. Code uses the VAE encoder mean, scaled by 0.18215 marigold_depth_pipeline.py. The paper does not state the scale.
• Decoding. Code clamps the decoded value to [-1, 1] and maps it to [0, 1] before averaging marigold_depth_pipeline.py. The paper does not describe this step.
• Model weights license. The code is Apache-2.0 LICENSE.txt. The model weights are under OpenRAIL++-M LICENSE-MODEL.txt. Under that license, offering the model as a hosted service counts as distribution LICENSE-MODEL.txt. This card covers the code only, not the weights.

Items that match: the effective batch of 32 with 16 accumulation steps §4.1 train_marigold_depth.yaml, and the per-image depth normalisation §3.3 depth_transform.py.

Pitfalls

• Scale and shift are unknown. The output is affine-invariant, not metric. Scoring aligns it to ground truth by least squares §4.2. For in-the-wild images the authors estimate scale, shift and intrinsics by hand, or read them from EXIF app. A.4. Do not read the raw output as metres.
• Runs differ. Each run gives a different depth map §3.4. A single run is less accurate than the 10-run ensemble tab. 1. Multi-resolution noise in training reduces this variance tab. S2.
• Cost is high. The paper uses 50 steps for each of 10 runs §4.1, which is about 500 U-Net passes per image. The authors call the approach slower than feed-forward estimators app. B.5. Code also runs a CPU BFGS alignment for the ensemble ensemble.py.
• Fewer steps may be fine, but unverified here. The paper reports saturation at about 10 steps fig. 7. The repo's eval script uses 1 step with its own checkpoint 11_infer_nyu.sh. The paper's numbers were not reproduced for this card.
• Normalisation clips outliers. The 2nd and 98th percentiles are taken over pixels with depth above zero, and values outside the range are clipped depth_transform.py. The paper does not discuss how this affects scenes with extreme depth.
• DIODE is weak. The paper's "most cases" claim §4.2 hides this. On DIODE, Marigold's AbsRel is 30.8 against 18.2 for DPT tab. 1.
• Ablations are not test results. Tables 2 and 3 use validation splits §4.3. Table 2's best row (5.6 / 96.5) is close to, but not the same as, the ensemble row in Table 1 (5.5 / 96.4). The paper does not say which setup the ablations used.
• Synthetic-only data has limits. Virtual KITTI alone gives NYUv2 AbsRel 13.9 tab. 3. The mix matters app. B.4. The paper expects synthetic-to-real gaps to limit generalisation in some cases §3.3.
• Training budget. The 2.5-day figure is for one RTX 4090 §4.1. The code sets 30,000 iterations with a comment that about 20k is enough train_marigold_depth.yaml.
• Weights are not covered by the code license. OpenRAIL++-M applies to the weights LICENSE-MODEL.txt. Check it before serving or redistributing any checkpoint.

Lineage

• Builds on: Stable Diffusion v2, a latent diffusion model, as its backbone §4.1. It uses the DDPM training objective (arXiv:2006.11239) eq. 2, DDIM sampling (arXiv:2010.02502) §3.4 and the v-objective (arXiv:2202.00512) §4.1. Multi-resolution noise follows Whitaker (2023), cited without an arXiv id §3.3.
• Compared against: DiverseDepth, MiDaS, LeReS, HDN, DPT and Omnidata tab. 1.
• Superseded by: no card for it is in the wiki. The official repository's default checkpoint is prs-eth/marigold-depth-v1-1 infer.py.