ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction

In plain words

Teams that build self-driving systems want to replay recorded drives in a simulator, so a virtual car can take a slightly different path and see what its cameras would have shown. The usual way to build such a replay is to fit a 3D model of each scene separately, which takes a long time per scene. ReconDrive instead trains one network that has already learned to read 3D shape from images, and asks it to produce the whole scene in a single forward pass.

The scene is stored as a cloud of small soft 3D blobs (Gaussians). Each blob has a position, an orientation, a size, an opacity and a colour. The network looks at two moments of a drive, seen from six cameras, and predicts the blobs for both. Parked background and moving cars are handled differently: the moving cars are located in the images and each gets one velocity, so their blobs slide along a straight line over the short interval. The two sets of blobs are then brought into one coordinate frame and can be drawn from new camera positions.

The authors' claim is that this gets close to slow per-scene fitting in quality on their benchmark while being much faster. It relies on calibrated cameras and on vehicle positions over time, which the released code takes from the dataset's annotations rather than from the images alone.

TL;DR

• ReconDrive uses the VGGT 3D foundation model as a frozen backbone with LoRA adapters, plus two heads that predict Gaussian centres and other Gaussian attributes §3.2 app. B.
• Each temporal segment is built from two context frames. Background Gaussians stay still; vehicle Gaussians move at a constant velocity inside the segment §3.1 §3.3 eq. 4.
• Author-reported on nuScenes (14 validation scenes): scene reconstruction PSNR 32.66, SSIM 0.9589, LPIPS 0.0618, against the best per-scene baseline at PSNR 29.58. Inference takes about 15 s per scene, against about 30 minutes for per-scene methods in the text tab. 1 §4.2.
• The released code departs from the paper in several places: the velocities come from ground-truth boxes, SAM 2 is prompted with those boxes, the loss weights differ, and ego poses are required. Read "Paper vs code" before reimplementing.

Problem

Closed-loop evaluation of driving policies needs photorealistic re-rendering of logged scenes from new viewpoints §1. Per-scene 4D Gaussian splatting methods give good quality but need costly iterative optimisation for every scene, which the authors say does not scale to large urban data §1 §2. Earlier feed-forward methods are fast but lose photometric detail, and a static 3D foundation model does not represent moving traffic §1 §3.1. Applying VGGT directly to driving data also gives point maps with the wrong metric scale, which the authors fix with calibration app. C fig. 5 fig. 6.

Assumptions stated or implied by the method:

• six calibrated cameras per frame, with intrinsics and extrinsics §3.1 app. B;
• two context frames per segment, 0.5 s apart, from the 12 Hz key frames §4.1 app. B;
• ego-vehicle poses for both frames, used to move the second frame's Gaussians into the first frame's coordinates §3.3;
• vehicle boxes with velocities for the moving Gaussians §3.3.

Method

Scene model. Scenes are cut into temporal segments [T_s, T_{s+1}], and each segment gets its own 4D Gaussians §3.1. A Gaussian has a centre \mu_i(t), a rotation R_i, a scale s_i, an opacity \alpha_i and spherical-harmonic colour coefficients c_i §3.1 eq. 1. Static Gaussians keep their centre fixed. Dynamic Gaussians are assumed to move linearly inside a segment with a regressed velocity §3.1 eq. 2. Section 3.3 says the velocity is instead computed from annotated boxes; see "Paper vs code" item 1.

Backbone §3.2. Each image is resized and cut into 14-pixel patches. A DINOv2 encoder produces 1024-dimensional tokens. The tokens of all 2 \times O images (two frames, O = 6 cameras) go through 24 alternating-attention layers. One kind of layer attends within each frame, the other across all views and frames. The VGGT weights are frozen, and LoRA adapters (rank 8, scale 32) are added to the backbone §3.2. The code puts LoRA on the attention and MLP linear layers recondrive_model.py.

Centre head (GCPH) §3.2. A DPT head upsamples the fused features to the resized image resolution. A 3×3 convolution gives a depth per pixel. Each depth value is unprojected with the camera's intrinsics and extrinsics into the ego-vehicle frame, which gives the Gaussian centre for that pixel §3.2.

Attribute head (GPPH) §3.2. A second DPT head upsamples the same features, and the resized input image is concatenated before a convolution. The authors say the raw image is needed to keep high-frequency colour and texture §3.2.

Vehicle motion §3.3. Vehicle masks come from SAM 2 §3.3. For each vehicle, the authors take the displacement of its box centre between the two frames, both expressed in the ego frame of frame T_s, and divide by the time between the frames §3.3 eq. 3. Every pixel inside a vehicle's mask gets that velocity, so the output is a per-pixel velocity map §3.3 fig. 2.

Segment composition §3.3. The Gaussians of frame T_{s+1} are moved into the ego frame of T_s with the ego-pose transform. This applies to centres, rotations and colour coefficients. Then the moving Gaussians are shifted back by their velocity to time T_s. The paper notes that these two steps cannot be swapped, because the velocity is expressed in the frame-T_s ego frame §3.3. The two sets are concatenated, and at rendering time a Gaussian at time t sits at its centre plus velocity times (t - T_s) §3.3 eq. 4.

Training losses §3.4 app. B.

• Rendering loss: a perceptual term plus an L2 term, against ground-truth images of the rendered frames eq. 5.
• Projection loss: the source frame is warped into the target frame with the predicted depth and the target's camera pose, and compared with a masked L1 + SSIM loss eq. 6 app. B fig. 4.
• Norm loss: a penalty on the mean scale norm and on the mean absolute opacity eq. 7.
• Total: the sum of the three eq. 8.

Training schedule §4.2 app. B. Stage 1 trains on single frames. Stage 2 fine-tunes on two-frame input. Training clips are 6 frames (0.5 s), with the first and last frames as context. Rendered frames are drawn from the clip at random, with the probabilities listed in "Hyperparameters" app. B.

Pseudocode. Forward pass for one segment, as the method describes it §3.1 §3.2 §3.3. It does not include the training losses.

# ReconDrive forward pass for one segment [T_s, T_s+1]  [§3.1]
# Input:  images (2, O, 3, H, W), two context frames, O = 6 cameras
#         intrinsics K and camera-to-ego extrinsics per camera
#         ego poses of both frames; 2-D boxes and velocities per vehicle
# Output: one set of Gaussians in frame T_s's ego coordinates

def reconstruct_segment(images, K, cam_to_ego, ego_poses, boxes, velocities):
    # Shared backbone over all 2 x O images, LoRA-adapted  [§3.2]
    tokens = backbone(images)                 # (2, O, HW, 1024)
    # Depth per pixel from the centre head  [§3.2]
    depth = centre_head(tokens)               # (2, O, H, W)
    # Centres: unproject each pixel with calibration  [§3.2]
    centres = unproject(depth, K, cam_to_ego) # (2, O, H, W, 3)
    # Rotation, scale, opacity, colour; the head also sees the raw image  [§3.2]
    rotation, scale, opacity, sh = attribute_head(tokens, images)
    # Vehicle masks from SAM 2, given the annotated boxes  [§3.3]
    masks = segment_vehicles(images, boxes)   # (2, O, H, W)
    # One velocity per vehicle, shared over its mask  [§3.3] [eq. 3]
    flow = velocity_map(masks, velocities)    # (2, O, H, W, 3)
    # Move frame T_s+1 into frame T_s's ego frame  [§3.3]
    centres_next = to_ego(centres[1], ego_poses)
    # Shift its moving Gaussians back to time T_s  [§3.3] [eq. 4]
    centres_next = centres_next - flow[1] * interval  # interval in s
    # Rotations and colour coefficients use the same ego transform  [§3.3]
    # Concatenate both frames into one 4D Gaussian set  [§3.3]
    return concat(centres[0], centres_next), rotation, scale, opacity, sh

Key equations, as written in the paper (the paper's numbering is kept; the projection loss also appears in the appendix as eq. 9).

\mu_i(t) = \mu_{i,\text{init}} + v_i\,(t - T_s) eq. 2

v = \frac{\text{displacement}}{T_{s+1} - T_s} eq. 3

\mu_i(t) = \mu_i(T_s) + V(i)\,(t - T_s), \quad t \in [T_s, T_{s+1}] eq. 4

\mathcal{L}_{\text{render}} = \lambda_{\text{percep}}\,\mathcal{L}_{\text{percep}} + \lambda_{l2}\,\mathcal{L}_{l2} eq. 5

\mathcal{L}_{\text{project}} = \mathcal{L}_{\text{masked}}\left(\lambda_{l1}\,\mathcal{L}_{l1} + \lambda_{\text{ssim}}\,\mathcal{L}_{\text{ssim}}\right) eq. 6

\mathcal{L}_{\text{norm}} = \lambda_{\text{scale}}\,\mathbb{E}\left[\lVert \text{scale}_{\text{maps}} \rVert_2\right] + \lambda_{\text{opacity}}\,\mathbb{E}\left[\lvert \text{opacity}_{\text{maps}} \rvert\right] eq. 7

\mathcal{L} = \mathcal{L}_{\text{render}} + \mathcal{L}_{\text{project}} + \mathcal{L}_{\text{norm}} eq. 8

Hyperparameters

Values from the paper unless marked "config" or "code". Where the paper and the code differ, both are listed; see "Paper vs code".

Results

Author-reported. Setup: nuScenes, 700 training scenes, 14 validation scenes, rendering at 518 × 280 §4.1 app. A tab. 1. Per-scene baselines were run by the authors through an adapted DriveStudio code base, for 30,000 iterations per scene §4.1. The feed-forward baseline DrivingForward was given several frames as input §4.1.

Scene reconstruction and novel-view synthesis tab. 1:

3D perception on novel-view renders, with UniAD, offsets 0 to ±3 m tab. 2:

Temporal-input ablation, single frame versus two frames tab. 3:

Paper vs code

Official code: TuojingAI/ReconDrive at commit d2bc397b724d6cc021da22f8f57ad6af1cc53e3c (shallow clone, HEAD). The paper's front matter and the README both link it README.md. The code is also described as a "prototype" and only the base model is released (README, TODO list). The released checkpoints are a stage-1 and a stage-2 checkpoint [docs/TRAIN_EVAL.md#L7-L10]. Discrepancies between the paper and the code at this commit, each with both citations:

1. Velocity source. The paper says the dynamic-object motion is regressed §3.1 but computes it from annotated 3-D boxes between the two frames §3.3. The code has no velocity regression. The dataset builds each vehicle's velocity from its annotated boxes, in the ego frame of frame T_s, over context_span / 12 seconds vggt4dgs_dataset.py vggt4dgs_dataset.py. The model stores those velocities as they are recondrive_model.py. The track head that would have predicted flow is disabled recondrive_model.py recondrive_model.py.
2. Vehicle masks are prompted with ground-truth boxes. The paper says SAM 2 extracts instance regions of traffic participants §3.3. The code passes the annotated 2-D boxes to SAM 2 as prompts recondrive_model.py and states that no automatic detection is run recondrive_model.py.
3. Ground-truth poses and boxes at inference. The introduction describes vision-only inputs §1. The ego transforms between frames are computed from dataset ego poses vggt4dgs_dataset.py and the inference script keeps the vehicle annotations as input inference.py.
4. Rendering loss. The paper uses a VGG-19 feature loss and an L2 norm §3.4 §4.2. The code uses LPIPS with a VGG backbone recondrive_model.py, a squared error and weights 0.05 and 1.0 recondrive_model.py. It multiplies the sum by \lambda_{\text{gaussian}} = 2 recondrive.yaml recondrive_model.py. The effective weights are therefore 0.1 (LPIPS) and 2 (squared error), not the paper's 0.05 and 1.0.
5. Total loss. Equation 8 has three terms. The training step sums a rendering term, a depth term that is fixed at zero, the projection term and the norm term recondrive_model.py. The LiDAR depth loss and an edge-smoothness term (weights 0.001 and 0.1) only run in validation and test recondrive_model.py recondrive_model.py recondrive_model.py. The config says GT depth is not used in training recondrive.yaml.
6. Masked loss keeps invalid pixels. The appendix says invalid or occluded pixels are suppressed by multiplying with the mask app. B. The masked-mean helper adds eps = 0.1 to the mask, so masked-out pixels keep weight 0.1 loss_util.py recondrive_model.py.
7. Projection loss normalisation. The code matches the warped image's mean and standard deviation to the target's over the overlap before computing the loss recondrive_model.py recondrive_model.py. The paper does not mention this step.
8. Projection loss definition. The appendix first defines a masked L1 loss, then a combined L1 and SSIM loss with the same name app. B. The code uses 0.85\,\mathcal{L}_{l1} + 0.15\,\mathcal{L}_{\text{ssim}} recondrive_model.py.
9. Depth range. The appendix says depth is clamped to 1.5 to 110 m app. B. The code maps the raw head output d to 1.5 + 108.5\cdot\sigma(\log d), a smooth squash rather than a clamp recondrive_model.py recondrive.yaml.
10. What is frozen. The paper says the VGGT weights are frozen and only LoRA is trained §3.2. The code freezes the aggregator apart from LoRA recondrive_model.py recondrive_model.py. The VGGT depth head and the new Gaussian head are left trainable recondrive_model.py recondrive_model.py. A helper that freezes everything except these heads exists but is never called recondrive_model.py.
11. Frame sampling. The appendix gives six probabilities for the first to fifth frame app. B. The code draws seven independent Bernoulli variables with probabilities 0.7, 0.3, 0.2, 0.1, 0.1, 0.05 and 0, over frame indices 0 to 5, which includes frame 0. It falls back to a categorical draw if nothing is selected, and caps the set at four frames recondrive_model.py.
12. Learning rate. The paper gives 2 \times 10^{-5} for stage 1 §4.2. The config sets 10^{-5}, which is multiplied by \log_2 of the GPU count when auto-scaling is on recondrive.yaml recondrive_model.py. The config uses eight GPUs recondrive.yaml, so the effective rate is 3 \times 10^{-5}.
13. Epochs. The paper gives 10 epochs for stage 1 and 2 for stage 2 §4.2. The config sets train_epoch: 2 recondrive.yaml, and the trainer reads that value trainer.py. The data block also has num_epochs: 10 recondrive.yaml. Only stage 2 is documented [docs/TRAIN_EVAL.md#L19-L20], and it starts from a stage-1 checkpoint train.sh. The stage-1 recipe is not in the repository.
14. Batch size. The paper gives batch 4 for stage 1 §4.2. The only config sets batch 2 recondrive.yaml.
15. Gradient clipping and schedule. Neither is in the paper. The code clips the gradient norm at 1.0 trainer.py and uses cosine warm restarts with T_0 = 5, T_{mult} = 2 recondrive_model.py.
16. Norm loss acts on activated maps. The norm loss is computed on the scale and opacity after their activations recondrive_model.py recondrive_model.py. The scale is 0.01\,\mathrm{softplus}(x), so it is always positive and small at start. The paper does not name the activations.
17. Spherical harmonics. The paper does not give a degree §3.1. The code uses degree 4 recondrive.yaml, which gives 25 bands recondrive_model.py. It also damps each higher band by 0.1 \cdot 0.25^{\text{degree}} recondrive_model.py.
18. Dormant ICP refinement. The code has a comment about ICP refinement of ego poses and velocities recondrive_model.py. The refined branch reads an ego_T_ego_refined output recondrive_model.py recondrive_model.py, but the training path never sets it. models/compare.py does set it, but the trainer imports only models/recondrive_model.py trainer.py. So no refinement runs. The paper does not mention refinement.
19. Per-camera masks. Each camera has a mask image, loaded as mask, that is used as the reference mask in the projection loss vggt4dgs_dataset.py vggt4dgs_dataset.py recondrive_model.py. The paper does not mention these masks, and the repository does not document what they show.
20. Augmentation. The loader can crop and jitter colours, but the config sets neither vggt4dgs_data_module.py. The paper does not mention augmentation.
21. Precision. The trainer runs in fp32 trainer.py, but the backbone runs under bf16 autocast recondrive_model.py. The paper says nothing about precision.

Pitfalls

• "Eight out of nine metrics" cannot be checked. The abstract and §4.2 say ReconDrive beats per-scene methods on eight of nine metrics. The paper never lists the nine. In tables 1 and 2 there are eight non-speed metrics, and ReconDrive is best on seven. The exception is novel-view LPIPS, where DeformableGS is lower tab. 1.
• The text and the table disagree on LPIPS. The reconstruction paragraph says DeformableGS leads on LPIPS §4.2. Table 1 shows OmniRe at 0.1577, below DeformableGS at 0.1610 tab. 1. The novel-view paragraph calls ReconDrive's LPIPS "comparable" to the best baseline, but 0.2591 against 0.2342 is a visible gap §4.2 tab. 1.
• "About 30 minutes" is a rounding of 23 to 46 minutes §4.2 tab. 1. The 15 s figure is with caching across adjacent segments, which share context frames §4.2. The hardware for the timing is not stated.
• Ablation rounding. The LPIPS change in the temporal ablation is printed as −0.058, but 0.2591 − 0.3177 = −0.0586 tab. 3.
• Non-uniform resize. The paper says it downsamples by 3 §3.2, but 1600 ÷ 3 is about 533 and 900 ÷ 3 is 300. The input is resized to 518 × 280 app. B: about 3.09 times smaller horizontally and 3.21 times vertically, so the aspect ratio changes slightly.
• Baselines are the authors' own runs. Per-scene methods were re-run on 12 Hz data with an adapted codebase, and DrivingForward was given several frames §4.1. The numbers are not the original papers' numbers.
• Small evaluation set. Results are on 14 validation scenes chosen to cover day and night, weather, manoeuvres and traffic density app. A §4.1.
• Perception numbers depend on UniAD's training views. UniAD was pre-trained on original images at 2 Hz, and the rendered views are fed to it at 2 Hz §4.1. Changes in the rendering pipeline therefore also change the perception scores.
• Closed-loop use needs annotations. Because the velocities and vehicle masks come from annotated boxes and the poses from annotated ego poses (Paper vs code items 1 to 3), the method cannot run on raw sensor data alone as the code stands. The paper does not discuss this limit.
• Linear motion only. The authors list non-rigid and non-linear motion as unsupported §5. They also list disocclusion holes that appear when a moved vehicle uncovers background, and Gaussian redundancy from simply concatenating pixel-wise outputs §5.
• The L1 opacity penalty does not push toward 1. The text says it drives opacities to 0 or 1 §3.4. An L1 penalty on opacity only pushes values toward 0; the bimodal result has to come from elsewhere.
• Depth is needed for the projection loss. The projection loss warps with the predicted depth of the target frame app. B recondrive_model.py. Errors in depth therefore reach the photometric loss directly.
• Metric scale. Raw VGGT point maps are off in scale by a global factor app. C fig. 5 fig. 6. The authors add calibration to the centre head to place Gaussians metrically §3.2 app. C; they do not show that this alone fixes the scale.
• Ambiguous squared norm. The paper writes an L2 norm for the rendering term §3.4. The code uses a squared error recondrive_model.py. The two give different gradients.

Lineage

• Builds on: VGGT, which supplies the backbone and DPT heads §3.2. The compiled bibliography gives no arXiv id for it. Not a card in this wiki.
• Builds on: LoRA, rank 8 and scale 32 §3.2. Not a card in this wiki.
• Builds on: SAM 2 (arXiv:2408.00714), used for vehicle masks §3.3. Not a card in this wiki.
• Builds on: DINOv2, the image encoder §3.2. Not a card in this wiki.
• Compared against: Street Gaussians, PVG, DeformableGS and OmniRe as per-scene baselines, and DrivingForward as the feed-forward baseline §4.1. Not cards in this wiki.
• Related feed-forward work: DUSt3R, FastVGGT, StreamVGGT and DriveVGGT §2.
• Superseded by: not known from wiki content.