Text-to-3D using Gaussian Splatting

In plain words

Making a 3D object from a text prompt usually means adjusting a 3D shape until a 2D image model is happy with its rendered views. Shapes made this way often grow extra faces, such as a second face on the back of a head, or they collapse into a messy form.

This paper stores the object as a cloud of small soft blobs (Gaussians) rather than as a hidden neural function. Each blob has an explicit position, so a 3D model can directly nudge the whole shape. The method first takes a rough shape from a text-to-point-cloud model and fixes the overall form with both 2D and 3D guidance. Then it only improves the surface look with the 2D image model, adding blobs where there are gaps and removing faint ones.

The payoff is an object with a sensible overall shape and fine detail such as feathers or thatch. The paper reports this takes about 40 minutes per asset on one GPU.

TL;DR

• Text-to-3D with 3D Gaussian Splatting, optimised by Score Distillation Sampling (SDS) from a 2D Stable Diffusion prior §4 §5.1.
• Geometry stage: 2D SDS plus a 3D SDS on Gaussian centres, using a Point-E text-to-point-cloud prior §4.1 eq. 3.
• Appearance stage: 2D SDS only, with position and opacity penalties and compactness-based densification §4.2 eq. 4.
• No quantitative metrics are reported; all comparisons are visual §5.2 fig. 2 fig. 5.
• Relevant if you want explicit 3D priors in SDS-based text-to-3D, or Gaussian Splatting as a generative representation.

Problem

• SDS methods with NeRF or DMTet representations often produce collapsed geometry and the Janus problem, and the implicit representations make it hard to add 3D priors §1.
• The original 3DGS densification rule (split Gaussians with a large view-space gradient) is unreliable under SDS: a small threshold over-densifies because SDS gradients are noisy, and a large one gives blurry results §4.2 §5.3 fig. 7.
• Baselines are slow: the paper says it matches Magic3D and Fantasia3D time of about 40 minutes per asset, while improving fidelity §5.2.

Method

Overview. The asset is a set of N anisotropic Gaussians, each with a centre (mean) p_i, a covariance from scale and rotation, a colour and an opacity. Rendering alpha-composites the projected Gaussians with the standard point-based volume rendering formula §3.2 eq. 2. The optimisation runs in three parts: initialisation §4.3, geometry optimisation §4.1, and appearance refinement with densification §4.2. The 2D guidance is Stable Diffusion v1.5 with view-dependent prompts §5.1 app. A. The 3D guidance is a Point-E point-cloud diffusion model §4.1 §4.3.

Initialisation §4.3. Initialise the Gaussians from a Point-E point cloud for the prompt. Users may supply a mesh or point cloud instead. Point clouds are reduced with farthest point sampling and meshes with uniform surface sampling §4.3. Colours are random, because Point-E colours hurt results in early experiments §4.3 app. A. Scales and opacities are fixed values and rotations are identity §4.3.

Geometry optimisation §4.1 eq. 3. Both guidance terms apply, and the 3D term changes only the positions. The 3D SDS uses Point-E as its noise predictor. Eq. 3 combines the 2D SDS term and the weighted 3D SDS term.

Appearance refinement §4.2 eq. 4. Only the 2D SDS term drives the colours and opacities. The positions are regularised. The loop densifies and prunes Gaussians, as in the pseudocode below.

# Initialise Gaussians from a Point-E cloud  [§4.3]
def initialise(prompt, num_points):
    # Point-E points for the prompt  [code: utils/initialize.py#L110-L114]
    points = point_e_sample(prompt, num_points)
    # Random colours, not the Point-E colours  [§4.3]
    colours = random_uniform(num_points, 3)
    # Fixed scale and opacity values; identity rotation  [§4.3]
    scales = fixed_scale
    rotations = identity_quaternions(num_points)
    opacities = fixed_opacity
    return points, scales, rotations, colours, opacities

# Geometry stage loss  [§4.1] [eq. 3]
def geometry_loss(gaussians, prompt):
    # Render from a random camera and take the 2D SDS term  [eq. 3]
    image = render(gaussians, random_camera())
    loss = 0.1 * sds_image(image, prompt)   # lambda_SDS = 0.1  [§5.1]
    # 3D SDS on the Gaussian centres only, with the point-cloud prior  [eq. 3]
    loss += 0.01 * sds_point_cloud(gaussians.positions, prompt)
    # lambda_3D = 0.01  [§5.1]
    return loss

# Appearance refinement loop  [§4.2] [eq. 4]
def refine_step(gaussians, prompt, step):
    # 2D SDS only; no 3D prior in this stage  [§4.2]
    image = render(gaussians, random_camera())
    loss = 0.1 * sds_image(image, prompt)   # lambda_SDS = 0.1  [§5.1]
    # Penalise position norms  [eq. 4]
    loss += 1.0 * sum_norm(gaussians.positions)    # lambda_mean = 1.0  [§5.1]
    # Opacity weighted by a stop-gradient distance term  [eq. 4]
    loss += 100.0 * sum_stopgrad_norm_times_opacity(gaussians)
    # lambda_opacity = 100  [§5.1]
    if step % 500 == 0:
        # Split Gaussians with mean view-space gradient above T_pos  [§5.1]
        split_by_position_gradient(gaussians, threshold=0.02)
    if step % 1000 == 0:
        # Compactness-based densification  [§5.1] [§4.2]
        compactness_densify(gaussians, K)
    if step % 200 == 0:
        # Remove Gaussians with opacity below alpha_min = 0.05  [§5.1]
        prune_low_opacity(gaussians, alpha_min=0.05)
    return loss

# Compactness-based densification  [§4.2] [fig. 4]
def compactness_densify(gaussians, K):
    # K nearest neighbours per Gaussian; K is not stated in the paper
    neighbours = k_nearest(gaussians.positions, K)
    for each Gaussian i and neighbour j in neighbours[i]:
        # Centre distance and unit direction from i to j
        gap = distance(centre_i, centre_j)
        direction = (centre_j - centre_i) / gap
        # Each Gaussian's radius along that direction  [§4.2]
        radius_i = surface_radius(i, direction)
        radius_j = surface_radius(j, -direction)
        # Only fill the gap if the two Gaussians do not touch  [§4.2]
        if radius_i + radius_j < gap:
            # New centre on the segment
            # [code: gs/gaussian_splatting.py#L646-L654]
            new_centre = centre_i + direction * (gap + radius_i - radius_j) / 2
            # New extent equals the residual  [§4.2] (code divides by 6)
            new_scale = gap - radius_i - radius_j
            add_gaussian(new_centre, new_scale, colour_i, rotation_i, opacity_i)

Notes on the pseudocode: the per-step schedule is taken from the paper's numbers §5.1. The paper does not state the start and end steps for densification or pruning, so the step % ... gates are a reading and not the paper's text. The shipped code differs from several lines; see Paper vs code.

Key equations (copied from the paper's notation, cited):

\nabla_\theta\mathcal{L}_{\text{geometry}} = \mathbb{E}_{\epsilon_I,t}\left[w_I(t)(\epsilon_\phi(x_t;y,t)-\epsilon_I)\frac{\partial x}{\partial\theta}\right] + \lambda_{\text{3D}}\,\mathbb{E}_{\epsilon_P,t}\left[w_P(t)(\epsilon_\psi(p_t;y,t)-\epsilon_P)\right] eq. 3

\nabla_\theta\mathcal{L}_{\text{refine}} = \lambda_{\text{SDS}}\,\mathbb{E}_{\epsilon_I,t}\left[w_I(t)(\epsilon_\phi(x_t;y,t)-\epsilon_I)\frac{\partial x}{\partial\theta}\right] + \lambda_{\text{mean}}\nabla_\theta\sum_i\lVert\mathbf{p}_i\rVert + \lambda_{\text{opacity}}\nabla_\theta\sum_i\mathtt{sg}(\lVert\mathbf{p}_i\rVert)\cdot o_i eq. 4

Here p_t and x_t are the noisy positions and rendered image, w are weighting functions, and sg is the stop-gradient eq. 3 eq. 4.

User-guided generation app. B. Initial points are sampled uniformly from the surface of the user's mesh. A relatively small learning rate for positions keeps the user's shape. The paper does not give the learning-rate value.

Hyperparameters

Results

Author-reported. The paper gives no quantitative metrics. Setup: text prompts with Stable Diffusion v1.5 for 2D guidance and Point-E for 3D guidance, one NVIDIA 3090, batch size 4 §5.1 app. A. Baselines DreamFusion, Magic3D and ProlificDreamer were run with Stable DreamFusion and threestudio, since their official code was not public at the time fig. 2 app. A. Fantasia3D was run from its official code app. A.

Paper vs code

Code: official repository gsgen3d/gsgen (MIT), commit e60994e557b685d5df57ec0bb48460782091ec4d. The repo is a later version and includes other guidance models. The configs disagree with the paper on several points.

• Densification schedule. Paper: split every 500 iterations, compactness every 1000 §5.1. Code: one densify period per config. The only enabled config that runs compactness is conf/shrink_then_densify.yaml, with period 1000, and the legacy path runs split and compactness together at that period shrink_then_densify.yaml gaussian_splatting.py. The config with period 500 and compactness (conf/renderer/debug.yaml) is disabled debug.yaml. I found no enabled config that reproduces the two-period schedule.
• Opacity and position penalties are means in code, sums in eq. 4. Eq. 4 sums over Gaussians. The code averages: torch.mean over the opacity term, and the uniform_l1 mean penalty also averages gaussian_splatting.py gaussian_splatting.py. With 1e5 to 1e6 Gaussians app. A, a literal sum and a mean differ by that factor at the same lambda.
• Mean penalty target. Eq. 4 penalises the norm of p_i, which is distance from the origin. The text says it penalises deviation from positions found in the geometry stage §4.2. The paper is internally inconsistent here. The code follows the origin reading gaussian_splatting.py.
• lambda_mean is not in any shipped config. The mean penalty only runs if the config defines penalty.mean, and none does. The term is therefore zero gaussian_splatting.py regular.yaml.
• lambda_opacity = 100 only in one renderer config. It is set in conf/renderer/regular.yaml. The default conf/base.yaml sets it to 0.0 base.yaml regular.yaml.
• 3D point-cloud SDS is off in the default config. auxiliary.enabled is False in conf/base.yaml. It must be turned on explicitly base.yaml trainer.py. When on, the weight schedule [2000, 0.01, 0, 2001] gives lambda_3D = 0.01 until step 2000, then 0 base.yaml misc.py. The paper does not state this boundary. The auxiliary config uses guidance scale 100 point_e.yaml. The paper does not state the Point-E guidance scale.
• Compactness details not in the paper. The shipped compactness config first divides all scales by 1.5 gaussian_splatting.py shrink_then_densify.yaml and uses K = 3 shrink_then_densify.yaml. The paper describes neither step.
• New Gaussian scale. Paper: the new Gaussian's radius equals the residual §4.2. Code: the per-axis scale is the residual divided by 6 gaussian_splatting.py.
• Possible duplicate Gaussians (by reading, not run). Each neighbour search runs from each side of a pair, and the midpoint formula is symmetric gaussian_splatting.py. A mutually close pair may therefore produce two Gaussians at the same location.
• Neighbour search. Paper: KD-tree §4.2. Code: pytorch3d knn_points ops.py. pytorch3d must be installed.
• Pruning interval. Paper: every 200 iterations §5.1. Code: conf/renderer/regular.yaml uses 100 regular.yaml. Its radius thresholds (1.0 and 0.6) are not in the paper.
• Densification window. conf/renderer/regular.yaml uses warm-up 2000 and end 4999. The paper does not state either regular.yaml. Its scale split threshold is 0.01, while conf/base.yaml uses 0.02 base.yaml. Neither value appears in the paper.
• SDS timestep range. The paper does not state it. conf/guidance/sd.yaml fixes the maximum at 0.98 sd.yaml. The default conf/base.yaml anneals the maximum from 0.98 to 0.5 between steps 2000 and 2001 base.yaml.
• Background. The appendix describes a learnable MLP background app. A. The default conf/base.yaml uses a random background base.yaml. conf/renderer/regular.yaml uses a fixed black background regular.yaml. An MLP background config exists in conf/renderer/mlp_bg.yaml.
• Initialisation prompt. The default init.prompt is "a human face" base.yaml. The README says to set it for each run [README.md]. The code also flips the Point-E cloud's x and y axes when facex is on initialize.py. The paper does not describe the flip.
• Opacity clamp. The appendix states [0.004, 0.99] app. A. I did not find it in the non-debug CUDA code.
• Consistent items. Lambda_SDS = 0.1, guidance scale 100, Stable Diffusion v1.5, batch size 4, and the 3D SDS applied to positions with colours detached all match the code base.yaml base.yaml base.yaml base.yaml point_e.py.

Pitfalls

• Unstated stage boundary. The paper never states when the geometry stage ends. The config uses step 2000 for the 3D term, the densify warm-up and related schedules base.yaml base.yaml. Pick one boundary and state it.
• Sum versus mean. Implementing eq. 4 literally as sums with lambda = 100 and lambda = 1.0 gives penalties much larger than the code's means. Decide which you mean and tune lambda to match.
• Janus is reduced, not removed. The paper says the Janus problem is not eliminated, and it shows failure cases where a second head appears on the appearance §6 app. C.
• Prompt complexity. Complex scene descriptions fail because of Point-E and the CLIP text encoder in Stable Diffusion §6. Stable Diffusion also fails on some spatial-relation prompts app. C.
• Guidance choice changes the look. DeepFloyd IF improves text alignment but works at 64 by 64 and looks blurrier. MVDream reduces Janus but ignores parts of complex prompts app. B.
• Densification thresholds are sensitive. Under SDS, a small T_pos over-densifies and a large one over-smooths §5.3 fig. 7. The paper gives one value and no sweep.
• Compute. About 40 minutes per asset on one 3090, with 1e5 to 1e6 Gaussians app. A. The paper gives no memory figure.
• Build and dependencies. The repository ships its own CUDA extension (gs/build.sh) and needs pytorch3d for the nearest-neighbour step build.sh ops.py. The paper says the authors re-implemented 3DGS with a plenoxel-style backward pass instead of using the official 3DGS code app. A, so the official 3DGS code will not reproduce this renderer's gradient order.
• Point-E is a rough prior. The paper reports good results even when Point-E gives only a rough shape app. B. It does not say how much rough is too much.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al.) for the representation and the splatting renderer §2.1 §3.2; DreamFusion for score distillation sampling §3.1; Point-E for the text-to-point-cloud prior §4.1; Stable Diffusion v1.5 for the 2D prior §5.1. None of these has a card here.
• Compared against: Magic3D §5.2 fig. 5; Fantasia3D, ProlificDreamer, DreamFusion §5.2 fig. 2. Also DreamGaussian, a concurrent image-to-3D method with Gaussian Splatting §2.3.
• Superseded by: not known from wiki content.