RealFusion: 360° Reconstruction of Any Object from a Single Image

In plain words

Give the method one photo of an object, with the background cut out, and it builds a 3D model you can look at from any side. A single photo says little about the parts you cannot see, so the method borrows what a text-to-image diffusion model knows about how objects look from other angles.

First it teaches the image model a new made-up word that stands for this one object. It does this with many randomly altered copies of the photo, so the word captures the object rather than the exact photo. Then it builds a 3D shape and trains it two ways at once. From the photo's own viewpoint, the shape must match the photo. From random other viewpoints, the image model must find the shape plausible when asked about that word. The result matches the photo and gives a reasonable guess for the hidden side. The guess is not guaranteed to be right: the back sometimes copies the front, and the shape can have floating bits or several faces.

TL;DR

RealFusion fits a radiance field, built on an Instant-NGP feature grid, to one RGBA input view. The extra signal comes from score distillation sampling (SDS) through Stable Diffusion on randomly placed views, with a prompt built from a token learned by single-image textual inversion. Smoothness and coarse-to-fine terms stabilise the surface. Author-reported, it beats Shelf-Supervised Mesh Prediction on mean F-score and CLIP similarity over 7 CO3D categories, but trails depth-supervised methods on PSNR in a favourable novel-view test. The official Apache-2.0 repository differs from the paper in several settings; read "Paper vs code" before reimplementing.

Problem

A single view leaves the 3D shape of the object largely unknown, and fitting a radiance field to one image overfits: a zero-loss solution exists that is not a sensible 3D model §1 §3.1. Training a 3D diffusion model would need 3D data that is scarce, so the method instead reuses a 2D diffusion model as a prior §1.

DreamFusion-style SDS can generate 3D from text. It needs a classifier-free guidance weight of 100 and produces the most typical object for a prompt, so its output is not a reconstruction of a given object §3.1. Applying the same idea to a reconstruction with a generic prompt fits the input view but shows a generic object from the back §3.2 fig. 7. The paper targets this gap.

The method assumes an RGBA input with an object mask, a single view with a standard camera, and no category-specific training or 3D supervision §1 §3.2 app. A.

Method

Overview. The radiance field has a fixed reconstruction camera. The input image supervises it there through an image term and a mask term. Random prior cameras supervise it through SDS, with a prompt that comes from single-image textual inversion, and through normal smoothness on the same prior view. The full objective is eq. 6 §3.2.

Radiance fields. A radiance field maps a 3D point to an opacity and a colour. A ray's pixel colour comes from the emission-absorption model in eq. 1, which is differentiable §3.1.

I(u) = \mathcal{R}(u;\sigma,c) = \sum_{i\in \mathcal{N}} (T_{i+1} - T_i)\, c(\bm{x}_i), \qquad T_i = \exp\!\Big(-\Delta\sum_{j=0}^{i-1} \sigma(\bm{x}_j)\Big) eq. 1

The reconstruction loss in eq. 2 is a squared image error over the training views. The paper states it as the squared norm of the difference, without a reduction. The single-image case uses one view with known camera §3.1.

\mathcal{L}_\text{rec}(\sigma,c;\mathcal{D}) = \frac{1}{|\mathcal{D}|}\sum_{(I,\pi)\in\mathcal{D}} \| I - \mathcal{R}(\cdot;\sigma,c,\pi) \|^2 eq. 2

Diffusion prior. A diffusion model is trained with the noise-prediction loss in eq. 3, where the noisy sample is built from a variance schedule §3.1.

\mathcal{L}_\text{diff}(\Phi;\mathcal{D}') = \frac{1}{|\mathcal{D}'|}\sum_{I\in\mathcal{D}'} \big\| \Phi\big(\sqrt{\bar{\alpha}_t} I + \sqrt{1-\bar{\alpha}_t}\,\epsilon, t\big) - \epsilon \big\|^2 eq. 3

SDS. SDS updates the field through the frozen denoiser, without the Jacobian of the denoiser. The paper states the timestep weight only as w(t), without a formula. The condition is the prompt embedding §3.1. Removing the Jacobian is stated to improve quality and reduce compute and memory §3.1.

\nabla_{(\sigma, c)} \mathcal{L}_\text{SDS}(\sigma,c;\pi,\bm{e},t) = \mathbb{E}_{t,\epsilon}\Big[ w(t)\, \big(\Phi(\alpha_t I + \sigma_t\epsilon; t,\bm{e}) - \epsilon\big) \cdot \nabla_{(\sigma, c)} I \Big], \quad I = \mathcal{R}(\cdot;\sigma,c,\pi) eq. 4

Single-image textual inversion. This is the core contribution. Instead of the alternative views the paper cannot get, it writes a prompt that stands for this specific object. The paper generates random augmentations g(I0) of the input and uses them as a mini-dataset. It then optimises the diffusion loss in eq. 3 with respect to one new token embedding, with all other embeddings and all model weights frozen. The prompt has the form "an image of a ". Unlike standard textual inversion, the paper uses one image and augmentations instead of several views §3.2. Initialisation of the token is random, manual or automatic (CLIP over WordNet nouns) app. A.

Coarse-to-fine grid. The field is an Instant-NGP model with 16 feature levels. For the first half of training only the lowest-resolution 8 levels are active. Then all 16 are used §3.2 app. A.

Normal smoothness. Normals are computed per ray and composited, then regularised in 2D rather than 3D. The paper's eq. 5 penalises the difference between the normal map and its Gaussian blur, with kernel size k = 9, and the blur is stop-gradient. The paper says normals may come from the density gradient or from finite differences, and it reports that finite differences worked well §3.2.

\mathcal{L}_\text{normals} = \big\| N - \operatorname{stopgrad}\big(\text{blur}(N, k)\big) \big\|^2 eq. 5

Mask. An off-the-shelf matting model supplies the object mask. The paper applies an L2 term between rendered opacity from the fixed camera and the mask §3.2.

Total objective. The reconstruction terms and the prior term are summed as in eq. 6. The paper uses L_image and L_mask there but defines L_rec and L_rec,mask in the text, so the card takes them to be the same terms §3.2.

\nabla_{\sigma,c}\mathcal{L} = \nabla\mathcal{L}_\text{SDS} + \lambda_\text{normals}\,\nabla\mathcal{L}_\text{normals} + \lambda_\text{image}\,\nabla\mathcal{L}_\text{image} + \lambda_\text{mask}\,\nabla\mathcal{L}_\text{mask} eq. 6

Prior-view details. Prior cameras and shading are sampled per step, with view-dependent prompts app. A:

• Radius, FOV and elevation are sampled as listed in Hyperparameters.
• Shading is albedo only for the first 1000 steps. After that the paper draws albedo, diffuse or textureless shading with probabilities 0.2, 0.4 and 0.4.
• The light is a point at the prior camera position plus Gaussian noise.
• The prompt gets a suffix from the prior camera's direction: "overhead view", "bottom view", "front view", "side view" or "back view". The paper's text labels the third azimuth band "bottom view", which is most likely meant to be "back view" (see Paper vs code).

Density bias and background. A Gaussian density blob at the origin helps early optimisation. The background is a weak two-layer MLP on view direction, so it cannot explain the object app. A.

Pseudocode. The step below is the paper's objective written as one optimisation step. Notation is our own. Symbols that are not stated in the paper are marked.

# One RealFusion optimisation step  [§3.2] [eq. 6]
# Inputs: field (radiance field), image (RGBA input view), step
# Output: scalar loss; gradients are accumulated on field

def train_step(field, image, step):
    # 1. Render the input view from the fixed camera, albedo only  [app. A]
    rgb, opacity = render(field, input_camera, shading="albedo")
    # Image term on object pixels, weight 5.0  [§4.1] [eq. 2]
    loss = 5.0 * mse(rgb * mask, image.rgb * mask)
    # Mask term on rendered opacity, weight 0.5  [§4.1] [app. A]
    loss = loss + 0.5 * mse(opacity, mask)

    # 2. Render a random prior view  [app. A]
    cam = sample_prior_camera()  # radius, elevation, azimuth, FOV
    shading = sample_shading(step)  # albedo for the first 1000 steps
    rgb_p, normals_p = render(field, cam, shading=shading)
    # Prior term: SDS through the frozen diffusion model  [§3.2] [eq. 4]
    prompt = view_prompt(cam)  # token plus view suffix  [app. A]
    loss = loss + sds(rgb_p, prompt, guidance=100.0)  # [§3.1] [app. A]
    # Normal smoothness on the prior view, weight 0.5  [§3.2] [eq. 5]
    blurred = blur(normals_p, kernel=9).detach()
    loss = loss + 0.5 * mse(normals_p, blurred)
    return loss

Single-image textual inversion runs before this step, as a separate stage:

# Single-image textual inversion  [§3.2] [app. A]
# Inputs: image (RGB, 512 px), init_emb (embedding of a starting word)
# Output: learned embedding for the new token <e>

def textual_inversion(image, init_emb, steps=3000):
    # 1. Only the new token's embedding is trainable  [§3.2] [app. A]
    emb = init_emb.clone().requires_grad_(True)
    for step in range(steps):
        # 2. Heavy random augmentation of the one image  [app. A]
        view = augment(image)
        # 3. Diffusion loss with the prompt "an image of a <e>"  [eq. 3]
        loss = diffusion_loss(view, prompt_with(emb))
        # 4. Adam update, batch of 16, lr 5e-4, weight decay 1e-2  [app. A]
        emb = adam_step(emb, loss, lr=5e-4, weight_decay=1e-2)
    return emb

Hyperparameters

Values are from the paper unless the Where column names code. "not stated" means the paper gives no value.

Results

Author-reported. Setting: CO3D §4.2, 7 categories, 3 random images each, one reconstruction per image, against Shelf-Supervised Mesh Prediction, which uses one pretrained model per category §4.2 tab. 1. F-score is for the shape, CLIP similarity for the appearance of novel views.

tab. 1. The paper says RealFusion is higher in aggregate, but its F-score is lower on Skateboard and Teddybear, and the CLIP scores tie on Skateboard. The caption says the metrics are averaged over three images per category, so the means are noisy.

Novel-view synthesis on the synthetic lego scene, using the SinNeRF test set of 60 views near the reference view. The paper stresses that this set favours depth-supervised methods and that RealFusion is not tailored to it app. D tab. 2.

tab. 2

Qualitative results are reconstructions from several viewpoints of in-the-wild and dataset images §4.3 fig. 4. Repeated runs from one input vary little at the front and a lot at the back §4.3 fig. 6.

Paper vs code

Code at code_commit (repository README calls it a refactor of the paper's code [README.md, "Code Overview"]). Each item cites the paper and the code.

• Camera replacement runs on 9 of 10 steps, and does not move the rays. Paper: every tenth iteration the prior camera is put near the reconstruction camera, with noise of variance 1 app. A. Code: the flag tests the remainder of the step count, so it is true on every step that is not a multiple of 10 trainer.py. In that branch the reconstruction batch is loaded but the rays still come from the synthetic camera, with noise of standard deviation 0.02 trainer.py trainer.py options.py.
• Front prompt on 9 of 10 steps. The prompt is the view-dependent one only when the camera is not replaced, that is on steps that are a multiple of 10. Other steps use the front-view embedding trainer.py. The paper gives no such split. This is read from the code and was not run.
• View regions. Paper: front is azimuth within ±30°, side is ±30 to 90°, and "bottom view" is labelled for ±90 to 180° app. A. Code: azimuth is absolute, front covers [0°, 60°) only, back covers [180°, 240°), overhead covers elevation of at least 60°, and bottom covers elevation of at most −60° provider_synthetic.py options.py. The paper's bottom threshold is elevation below 0°.
• Elevation range of prior cameras. Paper: elevation uniform in degrees over [−10°, 90°] app. A. Code: the polar angle is uniform on [0°, 120°], which is elevation [−30°, 90°], and the non-uniform branch is used with probability 0.5 provider_synthetic.py provider_synthetic.py.
• Coarse-to-fine switch point. Paper: first half of training, i.e. about step 2500 of 5000 §3.2 app. A. Code: masking is lifted only after step 3000 options.py trainer.py.
• Entropy regulariser. Paper: the appendix gives a formula that is not the binary entropy, since its first term has the opposite sign, and gives no weight app. A. Code: binary entropy with weight 1e-4 trainer.py options.py. Paper eq. 6 omits the orientation and entropy terms.
• Optimiser. Paper: Adam, learning rate 1e-3, 5000 iterations app. A. Code: Adam with betas (0.9, 0.99) and epsilon 1e-15 main.py. Encoder parameters use 10 times the learning rate, and the MLPs use the base rate network_grid.py. The default option named "adamw" selects this same Adam branch options.py.
• Weight averaging. Code: an exponential moving average of weights with decay 0.95, updated once per epoch (100 steps) and used for evaluation options.py trainer.py trainer.py trainer.py. Paper: not stated.
• Loss reduction. Paper: squared norms with no reduction §3.2 eq. 2 eq. 5. Code: mean squared error over sampled rays and pixels trainer.py trainer.py. The weights 5.0, 0.5 and 0.5 are only meaningful with the code's reduction.
• Blur sigma. Paper: kernel size 9, sigma not stated §3.2. Code: kernel 9 with no sigma passed, so torchvision chooses it from the kernel size trainer.py.
• SDS timestep weight. Paper: w(t) with no formula §3.1. Code: 1 minus the cumulative alpha at t, and t drawn from [20, 980] of 1000 steps sd.py sd.py sd.py.
• Maximum grid resolution. Paper: 2048 app. A. Code: the grid is built with the maximum resolution times the scene bound, 2048 × 0.75 = 1536, and a hash table of size 2^16 network_grid.py options.py.
• Reconstruction camera and resolution. Paper: not stated. Code: fixed field of view at the midpoint of [40°, 70°], i.e. 55°; the input is resized bilinearly to 128 px and 4096 random rays are used per step provider_image.py options.py options.py.
• Real-camera noise. Code only: noise of standard deviation 0.001 on the reconstruction rays, annealed to zero over training trainer.py options.py. Not in the paper.
• Light. Paper: a point light app. A. Code: a light direction, taken as the normalised camera origin plus Gaussian noise renderer.py. The diffuse term is 0.1 ambient plus 0.9 times the clamped cosine network_grid.py.
• Textual-inversion learning rate and batch. Paper: batch 16, learning rate 5e-4 app. A. The README command uses batch 1 with 4 accumulation steps and --scale_lr, which multiplies the learning rate by accumulation, batch and process count. That gives effective batch 4 and learning rate 2e-3 [README.md#L123-L133] textual_inversion.py.
• Textual-inversion optimiser and augmentation. Paper: Adam with weight decay 1e-2 and a random-resized-crop scale of (0.70, 1.30) app. A. Code: AdamW with weight decay 1e-2, betas (0.9, 0.999) and epsilon 1e-8 textual_inversion.py textual_inversion.py. The released augmentation uses scale (0.85, 1.15) plus extra resize and crop steps textual_inversion.py.
• Prompt templates. Paper: a template like "an image of a " §3.2. Code: one of a set of ImageNet-style templates, such as "a photo of a {}" textual_inversion.py textual_inversion.py. The default reconstruction prompt is "A high-resolution DSLR image of a " options.py.
• Diffusion model version. Paper: "open-source Stable Diffusion", version not stated §4.1. Code: runwayml/stable-diffusion-v1-5 options.py.
• Pose angle in the README. The README gives the default as 80 degrees, but the option is 75 options.py [README.md]. The paper's 15° above the plane matches 75 from vertical app. A.
• Matched settings. These agree between paper and code: radius range [1.0, 1.5], FOV range [40°, 70°], bound 0.75, density blob (5, 0.2), 16 levels with 2 features each, MLP 3×64, 96 px prior render, timestep range [0.02, 0.98], guidance 100, image weight 5.0, mask weight 0.5, normal weight 0.5, orientation weight 0.01, 5000 iterations, learning rate 1e-3 and albedo for the first 1000 steps options.py trainer.py.

Not checked: the CUDA raymarching and hash-grid kernels, the diffusion wrapper's loading code, and the runtime figure. No training was run.

Pitfalls

• Ambiguous prompt split. The paper's prompt suffix is clear in intent but not in its angle bands, and its third label is a probable typo. The code differs from the appendix here and in when the prompt is chosen (see Paper vs code), so implement from the code path you decide to keep.
• Mixed-up losses. Eq. 6 names L_image and L_mask, while the text defines L_rec and L_rec,mask. Their reduction is not given, so the weights 5.0 and 0.5 depend on it.
• Coarse-to-fine is central to smoothness. Without masking, the paper reports small irregularities on the surface §3.2 fig. 8. Changing the switch step changes how long the coarse grid is trained.
• Classifier-free guidance is high on purpose. Guidance of 100 is needed for the SDS signal. The appendix says values below 30 produced many geometric deformities app. A. Lower values change the result a lot.
• Failure modes. The paper reports neural fields without a clear surface, which are common with a CLIP prior and occasionally with Stable Diffusion app. F. It also reports floating parts in front of the camera, which are a big issue without image-specific prompts and are reduced but not removed with them app. F. The Janus problem, with two or more faces, persists even with view prompts app. F fig. 11. The authors also report that the model sometimes fails to converge, or copies the front view to the back §4.3 fig. 11.
• Textual inversion is the main lever. The ablation shows that without it the back often becomes a generic member of the category §4.4 fig. 7. The appendix says heavy augmentation is what makes single-image inversion work app. A. Initialisation has several options, so the result depends on the choice.
• Render resolution. The prior renders at 96 px. The paper reports no gain from higher resolution than 96 or 128 px app. G. The reconstruction term runs at 128 px in code options.py.
• Gradient path. Gradients from SDS flow through the VAE encoder, not through the UNet app. A sd.py.
• Per-image settings. The paper uses the same hyperparameters for all experiments, with a footnote for a few scenes where the camera angle was above 15 degrees §4.1. Appendix A gives 40 degrees for those scenes. The footnote says 30 or 40 degrees. Both are reported; neither is given per scene.
• Small comparison. The CO3D comparison uses 3 images per category and 7 categories, so the means rest on about 21 reconstructions §4.2 tab. 1.
• Costs the abstract hides. Textual inversion is a separate stage and, per the README, takes about an hour on a V100 [README.md]. Reconstruction takes about 45 minutes on one V100 app. A. Memory is not stated.
• Paper's own negative results. Using the diffusion model as a reconstruction loss, 3D normal smoothing, monocular depth, LPIPS and SSIM losses, higher render resolution, and DINO features did not clearly help, so the paper left them out app. G.

Lineage

• Builds on: DreamFusion (Poole et al., 2022) for SDS and the guidance scale of 100 §3.1. Textual inversion (Gal et al., 2022), extended here to one image with heavy augmentation §3.2. Instant-NGP (Müller et al., 2022) as the radiance field backbone §3.2. Stable Diffusion (Rombach et al., 2022) as the prior §4.1.
• Compared against: Shelf-Supervised Mesh Prediction (Ye et al., 2021) on CO3D §4.2 tab. 1. PixelNeRF, DietNeRF, DS-NeRF and SinNeRF on the lego test set app. D tab. 2. CLIP as a prior, in an ablation §4.4 fig. 10.
• Superseded by: not known from content already in this wiki.