2D Gaussian Splatting for Geometrically Accurate Radiance Fields

In plain words

3D Gaussian Splatting rebuilds a scene from photos as millions of small, fuzzy, coloured blobs that can be drawn very fast. The pictures look great, but the blobs have no surface: each one is a soft 3D cloud, and where a camera ray "hits" it depends on the viewing angle. Extracting a clean 3D mesh from such a cloud gives noisy, bumpy geometry.

2D Gaussian Splatting flattens every blob into a thin elliptical disk, like a sticker floating in space. A disk has a clear front, back and normal direction, and a camera ray hits it at exactly one point no matter where the camera stands. So the depth you render from one view agrees with the depth from another view.

Rendering the disks is done by intersecting each pixel's ray with the disk plane, not by the usual approximate projection, so the result stays correct in perspective. Two extra training penalties push the disks to bunch up along each ray and to lie flat against the surface implied by the rendered depth. After training, depth maps from all cameras are fused into a mesh.

The result keeps most of 3D Gaussian Splatting's image quality and speed, while producing far cleaner surfaces than it on the DTU benchmark, at a fraction of the training time of neural implicit-surface methods.

TL;DR

2DGS represents a scene as oriented planar 2D Gaussians (surfels) with a center, two tangent vectors and two scales, so each primitive has a well-defined normal §4.1. Pixels are evaluated by an explicit ray-splat intersection computed from two transformed homogeneous planes, with an object-space low-pass filter for edge-on splats §4.2. Training adds a depth distortion loss and a normal consistency loss to the 3DGS photometric loss §5. Meshes come from TSDF fusion of rendered median depth maps §6.1. Author-reported DTU Chamfer distance 0.80 (3DGS 1.96, SuGaR 1.33) at 10.9 min training, while Mip-NeRF 360 rendering quality is slightly below 3DGS tab. 1 tab. 4.

Problem

3DGS models scenes with volumetric 3D Gaussians and renders them with an affine (local linear) approximation of the projection §3. For surface reconstruction this has four problems the paper names §3: volumetric blobs conflict with thin surfaces; 3DGS has no native surface normal; the plane at which a 3D Gaussian is evaluated changes with viewpoint, so geometry is not multi-view consistent fig. 2; and the affine projection is only accurate near each Gaussian's center. Meshes extracted from 3DGS are therefore noisy fig. 5. Neural SDF methods (NeuS, VolSDF) give good surfaces but take more than 12 hours per DTU scene tab. 1. Assumptions: posed multi-view RGB images plus a sparse SfM point cloud for initialisation §4.1 §5; surfaces are assumed fully opaque §7.

Method

Primitive §4.1. Each splat has center \mathbf{p}_k, two orthogonal unit tangent vectors \mathbf{t}_u, \mathbf{t}_v, scales (s_u, s_v), opacity \alpha and spherical-harmonics colour. The normal is \mathbf{t}_w = \mathbf{t}_u \times \mathbf{t}_v; \mathbf{R} = [\mathbf{t}_u, \mathbf{t}_v, \mathbf{t}_w] and \mathbf{S} is a 3\times3 diagonal matrix whose last entry is zero. A point on the splat plane is

P(u,v) = \mathbf{p}_k + s_u \mathbf{t}_u u + s_v \mathbf{t}_v v = \mathbf{H}(u,v,1,1)^\top eq. 4

\mathbf{H} = \begin{bmatrix} s_u \mathbf{t}_u & s_v \mathbf{t}_v & \mathbf{0} & \mathbf{p}_k \\ 0 & 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} \mathbf{R}\mathbf{S} & \mathbf{p}_k \\ \mathbf{0} & 1 \end{bmatrix} eq. 5

and the Gaussian value in local coordinates is

\mathcal{G}(\mathbf{u}) = \exp\left(-\frac{u^2+v^2}{2}\right) eq. 6

In code, the rotation is stored as an unnormalised quaternion (normalised on read), scales are stored as log values and passed through exp, opacity through a sigmoid gaussian_model.py.

Ray-splat intersection §4.2. With \mathbf{W} the 4×4 world-to-screen transform, a screen point is \mathbf{x} = (xz, yz, z, z)^\top = \mathbf{W}\mathbf{H}(u,v,1,1)^\top eq. 7. Inverting \mathbf{W}\mathbf{H} is unstable when a splat is seen edge-on, so instead the pixel ray is written as the intersection of the x-plane \mathbf{h}_x = (-1,0,0,x)^\top and y-plane \mathbf{h}_y = (0,-1,0,y)^\top, and both planes are moved into the splat's uv frame. Planes transform with the inverse transpose, so no inverse is needed §4.2:

\mathbf{h}_u = (\mathbf{W}\mathbf{H})^\top \mathbf{h}_x \qquad \mathbf{h}_v = (\mathbf{W}\mathbf{H})^\top \mathbf{h}_y eq. 8

The intersection point satisfies \mathbf{h}_u \cdot (u,v,1,1)^\top = \mathbf{h}_v \cdot (u,v,1,1)^\top = 0 eq. 9, giving (superscripts index the 4 plane components; the third components are always zero) eq. 10:

u(\mathbf{x}) = \frac{h_u^2 h_v^4 - h_u^4 h_v^2}{h_u^1 h_v^2 - h_u^2 h_v^1} \qquad v(\mathbf{x}) = \frac{h_u^4 h_v^1 - h_u^1 h_v^4}{h_u^1 h_v^2 - h_u^2 h_v^1} eq. 10

Depth z of the intersection then follows from eq. 7 §4.2.

Low-pass filter for degenerate splats §4.2. An edge-on splat collapses to a line and can miss every pixel center. The value is lower-bounded by a screen-space Gaussian around the projected center \mathbf{c}:

\hat{\mathcal{G}}(\mathbf{x}) = \max\left\{ \mathcal{G}(\mathbf{u}(\mathbf{x})), \mathcal{G}\left(\frac{\mathbf{x}-\mathbf{c}}{\sigma}\right) \right\} eq. 11

with \sigma = \sqrt{2}/2 §4.2.

Rasterization §4.2. As in 3DGS: per-splat screen bounding box, sort by center depth, bin into tiles, then front-to-back alpha blending, stopping when accumulated opacity saturates:

\mathbf{c}(\mathbf{x}) = \sum_{i=1} \mathbf{c}_i\,\alpha_i\,\hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) \prod_{j=1}^{i-1} \left(1 - \alpha_j\,\hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))\right) eq. 12

The renderer additionally outputs depth, normal and distortion maps §6.1.

Depth distortion loss §5. With blending weight \omega_i = \alpha_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) \prod_{j<i}(1-\alpha_j\hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))) and intersection depth z_i:

\mathcal{L}_d = \sum_{i,j} \omega_i \omega_j |z_i - z_j| eq. 13

Unlike Mip-NeRF 360, gradients flow into z_i itself, pulling splats together along the ray §5. The implementation actually used is a squared version on NDC-mapped depth m = \mathrm{NDC}(z) with near 0.2 and far 1000, computed in one front-to-back pass app. A:

\mathcal{L} = \sum_{i=0}^{N-1} \sum_{j=0}^{i-1} \omega_i \omega_j (m_i - m_j)^2 = \sum_{i=0}^{N-1} \omega_i \left(m_i^2 A_{i-1} + D^2_{i-1} - 2 m_i D_{i-1}\right) app. A

where A_i = \sum_{j\le i}\omega_j, D_i = \sum_{j\le i}\omega_j m_j, D^2_i = \sum_{j\le i}\omega_j m_j^2 app. A.

Normal consistency loss §5. The surface is taken at the median intersection point \mathbf{p}_s where accumulated opacity reaches 0.5. The splat normal \mathbf{n}_i, flipped to face the camera, is aligned with the normal \mathbf{N} of the depth map:

\mathcal{L}_n = \sum_i \omega_i (1 - \mathbf{n}_i^\top \mathbf{N}) eq. 14

\mathbf{N}(x,y) = \frac{\nabla_x \mathbf{p}_s \times \nabla_y \mathbf{p}_s}{|\nabla_x \mathbf{p}_s \times \nabla_y \mathbf{p}_s|} eq. 15

with the gradients taken by finite differences of neighbouring depth points §5. The code uses central differences two pixels apart and leaves border pixels at zero point_utils.py.

Total loss §5:

\mathcal{L} = \mathcal{L}_c + \alpha \mathcal{L}_d + \beta \mathcal{L}_n eq. 16

\mathcal{L}_c is the 3DGS RGB loss (\mathcal{L}_1 plus D-SSIM). (This \alpha is a loss weight, not the splat opacity.)

Depths app. B. Mean depth z_\text{mean} = \sum_i \omega_i z_i / (\sum_i \omega_i + \epsilon); median depth z_\text{median} = \max\{z_i \mid T_i > 0.5\}, where T_i is the transmittance before splat i. If a ray never reaches 0.5 accumulated alpha, the last Gaussian is used.

Densification §6.1. 3DGS adaptive control, but the densification gradient is the gradient of the 3D center \mathbf{p}_k projected to screen space (2DGS has no 2D projected center in its backward pass). In code, a split samples new centers inside the splat's tangent plane (zero standard deviation along the normal) and divides scales by 1.6 for N=2 children gaussian_model.py.

Mesh extraction §6.1. Render (median) depth for all training views, fuse with Open3D TSDF, voxel size 0.004, truncation 0.02. Median depth beats expected depth and SPSR on splat centers tab. 5.

# 2DGS training step (one random training view)
# Input: camera, splats {center, quat, log_scale(2), opacity, sh}, gt_image
# Output: scalar loss

def render_pixel(pixel_x, pixel_y, sorted_splats, camera):
    # pixel ray as two homogeneous planes  [§4.2]
    plane_x = (-1, 0, 0, pixel_x)
    plane_y = (0, -1, 0, pixel_y)
    transmittance = 1.0
    outputs = zeros()
    # splats sorted by center depth, front to back  [§4.2]
    for splat in sorted_splats:
        # splat-to-screen matrix W H, no inversion  [eq. 5] [eq. 8]
        world_to_screen_splat = camera.full_proj @ splat.H
        plane_u = transpose(world_to_screen_splat) @ plane_x
        plane_v = transpose(world_to_screen_splat) @ plane_y
        # closed-form intersection in uv space  [eq. 10]
        u, v = solve_two_planes(plane_u, plane_v)
        g_surface = exp(-(u * u + v * v) / 2)
        # screen-space low-pass, sigma = sqrt(2)/2  [eq. 11]
        offset = (pixel_x, pixel_y) - project(splat.center)
        g_filter = exp(-squared_norm(offset) / (2 * sigma * sigma))
        g = max(g_surface, g_filter)
        alpha = splat.opacity * g
        weight = alpha * transmittance
        # depth of the ray-splat hit point  [eq. 7]
        depth = intersection_depth(world_to_screen_splat, u, v)
        # one-pass distortion with running sums A, D, D2  [app. A]
        ndc = ndc_depth(depth, near=0.2, far=1000)
        outputs.distortion += weight * (
            ndc * ndc * outputs.acc_w
            + outputs.acc_wm2
            - 2 * ndc * outputs.acc_wm)
        outputs.acc_w += weight
        outputs.acc_wm += weight * ndc
        outputs.acc_wm2 += weight * ndc * ndc
        # median depth: last hit while T > 0.5  [app. B]
        if transmittance > 0.5:
            outputs.median_depth = depth
        # normal faces the camera  [§5]
        normal = facing_camera(splat.t_w, camera)
        outputs.color += weight * splat.sh_color(camera)
        outputs.normal += weight * normal
        outputs.depth_sum += weight * depth
        transmittance *= 1 - alpha
        # stop once accumulated opacity saturates  [§4.2]
        if transmittance is saturated:
            break
    return outputs

def training_step(camera, splats, gt_image, alpha_dist, beta_normal):
    maps = render_all_pixels(camera, splats)
    # L1 + D-SSIM from 3DGS  [§5]
    loss_color = rgb_loss(maps.color, gt_image)
    # depth-map normal by finite differences  [eq. 15]
    points = unproject(camera, maps.median_depth)
    depth_normal = normalize(cross(grad_x(points), grad_y(points)))
    # weighted normal agreement  [eq. 14]
    loss_normal = mean(maps.acc_w - dot(maps.normal, depth_normal))
    loss_dist = mean(maps.distortion)
    # total objective  [eq. 16]
    return loss_color + alpha_dist * loss_dist + beta_normal * loss_normal

Hyperparameters

Results

Author-reported. DTU: 15 scenes, 800×600, COLMAP sparse init, Chamfer distance (lower is better) §6.2 tab. 1. Times are per scene on one RTX 3090.

Per-scene DTU Chamfer distance tab. 1:

DTU efficiency tab. 3 (PSNR on training views):

Tanks and Temples, F1 score (higher is better) tab. 2:

Mip-NeRF 360 novel view synthesis, 30k iterations for 3DGS, SuGaR and 2DGS; other baselines copied from their papers tab. 4:

Ablations on DTU, 15k iterations tab. 5:

Additional baselines on DTU, 30k iterations tab. 6:

The affine-approximation baseline is 3DGS rasterization with one scale fixed to 1e-6 app. C.

NeRF-Synthetic PSNR, no regularizers tab. 7:

Tanks and Temples PSNR tab. 8:

Mip-NeRF 360 per scene tab. 9:

Paper vs code

Checked at code_commit. The CUDA rasterizer lives in the submodule hbb1/diff-surfel-rasterization, pinned at commit e0ed0207; it was read on GitHub rather than cloned, so its findings below name files but carry no line citations.

• Distortion weight α. Paper: 1000 bounded, 100 unbounded §5. Code default is 0, so distortion is off unless passed [code: arguments/init.py#L86]. The DTU script passes 1000 dtu_eval.py. The Tanks and Temples script prints 100 (or 10 for Meetingroom and Courthouse) but does not pass it to the command it runs, so those runs train with 0 tnt_eval.py. The Mip-NeRF 360 script passes no value either m360_eval.py.
• Distortion form and NDC far plane. Eq. 13 uses |z_i - z_j| §5; app. A and the rasterizer use squared differences of NDC depth. The paper gives near/far 0.2/1000 app. A; the rasterizer's cuda_rasterizer/auxiliary.h sets near_n = 0.2, far_n = 100.0.
• Regularizer warm-up. Not in the paper. Code enables distortion after iteration 3000 and normal consistency after iteration 7000 train.py.
• Normal loss reduction. Eq. 14 sums \omega_i(1-\mathbf{n}_i^\top \mathbf{N}) per ray. Code renders the weighted normal sum \sum_i \omega_i \mathbf{n}_i, multiplies the depth normal by the detached accumulated alpha, and averages one minus their dot product over all pixels [code: gaussian_renderer/init.py#L141-L147] train.py. So the code computes 1 - A\sum_i \omega_i \mathbf{n}_i^\top \mathbf{N} with A = \sum_i \omega_i, while eq. 14 gives A - \sum_i \omega_i \mathbf{n}_i^\top \mathbf{N}. The two agree only where A = 1; on partly transparent pixels the code scales the normal term by A.
• Depth used for the normal target. Paper: median point §5. Code mixes expected and median depth by depth_ratio, default 0 (expected depth) [code: arguments/init.py#L69] [code: gaussian_renderer/init.py#L137-L144]. The DTU and TnT scripts set 1.0 (median); Mip-NeRF 360 uses the default dtu_eval.py m360_eval.py. The README recommends expected depth for unbounded scenes to reduce disk aliasing.
• TSDF truncation. Paper: voxel 0.004, truncation 0.02 §6.1. Scripts use voxel 0.004, truncation 0.016, depth cut-off 3.0 for DTU and the four object-centric TnT scenes, and voxel 0.006, truncation 0.024, cut-off 4.5 for Meetingroom and Courthouse dtu_eval.py tnt_eval.py. Only the function default matches the paper mesh_utils.py. The depth cut-off is not stated in the paper.
• Opacity pruning. Paper: below 0.05 "every 3000 step" §6.1. Code prunes below 0.05 at every densification call (every 100 iterations from 500 to 15000), and also prunes splats larger than 20 px on screen or 0.1× scene extent after the first opacity reset; opacities are reset every 3000 iterations train.py gaussian_model.py.
• Median depth. App. B defines the maximum z_i with T_i > 0.5. The rasterizer's forward.cu keeps the depth of the last splat processed while T > 0.5, in center-depth sort order, which is not necessarily the largest intersection depth.
• Rasterizer details not in the paper (from forward.cu / auxiliary.h of the submodule): per-splat alpha clamped to 0.99, splats with alpha < 1/255 skipped, termination when transmittance would drop below 1e-4, 3σ bounding radius, normals flipped toward the camera, back-face culling flag enabled.
• Mip-NeRF 360 resolution. Script trains outdoor scenes on images_4 and indoor scenes on images_2 m360_eval.py; the paper does not state the resolution.

Pitfalls

• Do not reproduce TnT or Mip-NeRF 360 with the official scripts and expect the paper's α. As shipped they train with zero distortion weight; pass --lambda_dist yourself. Which α produced tab. 2 and tab. 4 is therefore uncertain.
• Two different distortion losses. Eq. 13 (L1 on raw depth) is not what is implemented (squared NDC depth, app. A). Use the app. A form and the NDC mapping if you want the reported numbers; also note the near/far mismatch (paper 1000, code 100). The NDC mapping down-weights distant splats, so the far plane changes the effective weight.
• The normal loss needs median or expected depth chosen per scene type. Median depth gives better DTU meshes tab. 5 but the authors' code defaults to expected depth for unbounded scenes; the paper does not mention this switch.
• Warm-up matters. Enabling the normal loss from iteration 0 on random initial orientations is not what the code does; it waits until 7000.
• Ray-splat intersection is a division. Eq. 10's denominator h_u^1 h_v^2 - h_u^2 h_v^1 goes to zero for edge-on splats; the low-pass max in eq. 11 is what keeps such splats visible and differentiable. Do not drop it.
• Densification signal is approximate. 2DGS has no 2D-center gradient; the 3D center gradient projected to screen is used instead §6.1. In code this is a dummy zero tensor whose gradient the rasterizer fills [code: gaussian_renderer/init.py#L26-L31].
• Splits stay in-plane. New centers are sampled only along the tangent directions gaussian_model.py; sampling in 3D like 3DGS would scatter points off the surface.
• Mesh extraction needs a depth cut-off. Bounded TSDF fusion needs a scene-dependent maximum depth (3.0 or 4.5 in scripts); the paper does not state it. Unbounded scenes need the separate contraction-based extractor in the code mesh_utils.py.
• Camera model. The README warns the code supports only ideal pinhole cameras with the principal point at the image center.
• Known limits §7: semi-transparent surfaces (glass) fail because full opacity is assumed; densification favours texture-rich over geometry-rich regions; the regularizers trade image quality for geometry and can over-smooth. Holes can appear in very bright regions fig. 12.
• Image quality is slightly below 3DGS on Mip-NeRF 360 (outdoor PSNR 24.34 vs 24.64) tab. 4 and on NeRF-Synthetic (33.07 vs 33.32) tab. 7.
• License. The official code is under the Inria Gaussian-Splatting license, non-commercial research use only; reimplement rather than copy.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., arXiv:2308.04079): same tile rasterizer, SH colour, adaptive density control and RGB loss; primitives flattened to 2D and projection replaced by ray-splat intersection §3 §4.
• Builds on: surface splatting and perspective-accurate splatting (Zwicker et al.), hardware ray-splat intersection (Weyrich et al.), object-space low-pass filter (Botsch et al.) §4.2; distortion loss from Mip-NeRF 360 (arXiv:2111.12077) §5.
• Compared against: NeuS, VolSDF, NeRF, 3DGS, SuGaR (arXiv:2311.12775), Geo-Neus, Neuralangelo tab. 1 tab. 2.