Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos

In plain words

Forge4D takes a few videos of a person, filmed from several cameras whose positions were never measured. In one pass through a neural network it builds a 3D model that can be shown from new viewpoints and at moments between the recorded frames. Earlier methods either needed a slow fitting step for each scene, or could only show the frames that were recorded.

The model describes each frame as a cloud of small 3D blobs (Gaussians). A learned token passes information from frame to frame, so the blobs of neighbouring moments agree on scale and do not drift. A second output predicts how every blob moves toward the next frame. To show a moment in between, the blobs of the two neighbouring frames are moved part of the way, and the ones that one frame sees but the other hides are kept apart from the merged ones. The motion is not labelled in the training data. It is learned by rendering the moved blobs and comparing them with the real photos, and by matching against motion estimated by a separate optical-flow model.

The authors report better image quality than the methods they compare with, on their own test sets. Their speed figures are for their own hardware and setup.

TL;DR

Forge4D is a feed-forward network for dynamic human-object scenes. It predicts pixel-aligned 3D Gaussians per frame from four uncalibrated views, with a learnable state token that links frames, and predicts a dense 3D motion for each Gaussian §3.1 §3.2 §3.3 §3.4. The motion is trained with a self-supervised retargeting loss and an occlusion-aware flow loss against an optical-flow model §3.4. Novel times are shown by moving the Gaussians of two neighbouring frames to the requested time and fusing them §3.4. Author-reported: PSNR 29.82 novel-view on DNA-Rendering against 26.12 for the pose-free AnySplat tab. 1, and 29.04 novel-view-and-time against 18.03 for L4GM at 8 input frames tab. 2. The code is Apache-2.0 but no checkpoints are released, and several training details differ from the paper (see Paper vs code).

Problem

Dynamic human reconstruction from a few cameras usually needs calibrated cameras and a long per-scene optimisation over dense video §1. Feed-forward reconstruction models that predict 3D Gaussians exist, but they are static: they handle one timestamp at a time, and they cannot produce times between frames §2. Point-map models estimate poses and points from uncalibrated images, but their point clouds give weaker photorealistic rendering §1 §2.

Two practical problems come up when one simply stacks video frames into a single transformer. First, global attention over all frames runs out of memory and is slow §3.3. Second, the Gaussian scale drifts between timestamps §3.3. The authors also note that direct regression of 4D Gaussians tangles geometry and motion, and that there is no ground-truth dense motion to train on §3.1 §3.4.

Method

The model is trained in three stages §3.1 fig. 1. Stage 1 is a static, pose-aware 3D Gaussian reconstruction for one timestamp. Stage 2 adds the state token so that frames are processed in order. Stage 3 adds the motion head and the Gaussian fusion used for novel times. The backbone is VGGT, initialised from its pre-trained weights §4.1.

Stage 1: static Gaussians with a metric gauge (§3.2)

For each view the model predicts pixel-aligned maps for Gaussian position, opacity, colour, rotation and scale §3.2. The position maps are the points of the Gaussians, and a DPT head produces the maps §3.2.

The VGGT output is in normalised units, not metric units §3.2. The paper therefore computes a single scale factor, the metric gauge. For each input camera it takes the ratio of the translation length of the predicted pose to that of the ground-truth pose, and the gauge is the mean of these ratios §3.2. The novel-view ground-truth extrinsics are rescaled with it §3.2. The camera loss is given in equation 2 eq. 2. Photometric losses (L2, SSIM and LPIPS) are applied to the input views and to m held-out novel views §3.2.

# Stage 1 camera loss  [§3.2] [eq. 2]
# Input:  predicted and ground-truth rotations q, translations T, gauge p
# Output: scalar loss

def camera_loss(pred, gt, p_gauge, p_hat_gauge):
    # Rotation term over all n+1 cameras  [eq. 2]
    loss = sum_norm(gt.q - pred.q)
    # Translation direction, scale-free, over the n non-reference cameras
    loss = loss + sum_norm(unit(gt.T) - unit(pred.T))
    # Each per-camera ratio should match the mean gauge  [eq. 2]
    loss = loss + sum_abs(p_i - p_gauge)
    # The predicted gauge head should match the mean gauge  [eq. 2]
    loss = loss + abs(p_hat_gauge - p_gauge)
    return loss

The per-camera ratio p_i is defined as above, so the loss uses the mean gauge as the reference for each ratio §3.2.

Stage 2: streaming with a state token (§3.3)

To avoid global attention over all frames, the model splits the input into a frame-wise part and a temporal part §3.3. A learnable state token carries information from earlier frames §3.3. In each frame the frame tokens use the state as keys and values in a cross-attention layer, and the state uses the frame tokens as queries to update itself §3.3.

The gauge is extended over time: the gauge is the average of per-camera, per-timestamp ratios §3.3. The temporal camera loss in equation 3 supervises the temporal cross-attention with the same terms as before eq. 3.

Stage 3: dense motion, retargeting and fusion (§3.4)

The motion head works on the features of two neighbouring frames, t and t-1, and predicts two 3D motion fields per view: a backward field from frame t to t-1, and a forward field from t-1 to t §3.4. The motion block has as many attention blocks as the backbone §3.4.

A constant-velocity model moves the Gaussians of each frame toward a middle time t'. The displacement is the motion field scaled by the time distance §3.4 eq. 5. This is a first-order approximation, justified by the authors only for short time gaps §3.4.

The retargeting loss needs no motion labels. Frame t's Gaussians are moved to t-1 by the predicted backward motion, rendered, and compared with the render of frame t-1 §3.4 eq. 6.

The flow loss matches the predicted 2D scene flow with flow from SEA-RAFT §3.4. A cyclic-consistency weight that is near zero in occluded regions removes those pixels §3.4 eq. 7.

Gaussians that are occluded in one of the two warped frames are kept separately. The rest are merged by a two-layer MLP, F_θ, into one set §3.4 eq. 8. The fused result is supervised by a photometric loss at the novel time §3.4.

# Interpolation at novel time t_mid in (t-1, t)  [§3.4] [eq. 5] [eq. 8]
# Input:  Gaussians of frames t-1 and t, their backward/forward motions
# Output: Gaussians for t_mid

def interpolate(gauss_prev, gauss_cur, motion_fw, motion_bw, t_mid):
    # Move each frame's Gaussians by a constant-velocity displacement [eq. 5]
    moved_cur = move(gauss_cur, motion_bw, abs(t_mid - t))
    moved_prev = move(gauss_prev, motion_fw, abs(t_mid - (t - 1)))
    # Split off Gaussians whose dual-consistency distance exceeds tau  [§3.4]
    occluded_cur, visible_cur = split(moved_cur, tau=0.05)   # [app. C]
    occluded_prev, visible_prev = split(moved_prev, tau=0.05)
    # Merge the visible Gaussians of the two frames with the fusion MLP  [§3.4]
    fused = fusion_mlp(visible_cur, visible_prev)            # F_theta
    # Output: occluded parts unchanged, plus the fused part  [eq. 8]
    return concat(occluded_cur, occluded_prev, fused)

The full stage-3 objective combines the matching losses with the fusion loss, which is written as L_4D = L_matching + L_fusion §3.4. L_matching is the sum of the flow loss and the retargeting loss §3.4.

# Stage 3 objective  [§3.4] [eq. 6] [eq. 7]
# Input:  predictions and ground truth for one clip
# Output: scalar loss

def stage3_loss(pred, gt):
    # Retargeting: frame t moved to t-1 should render like frame t-1  [eq. 6]
    l_retarget = photometric(pred.retarget_render, gt.frame_prev)
    # Flow: predicted scene flow vs a pseudo-label from an optical-flow model
    # The cyclic-consistency weight removes occluded pixels  [eq. 7]
    l_flow = cyclic_weighted_l2(pred.scene_flow, gt.flow, pred.cycle_weight)
    # Fusion: the rendered novel-time frame should match the real one  [§3.4]
    l_fusion = photometric(pred.novel_time_render, gt.novel_time)
    return l_retarget + l_flow + l_fusion

Equations used above, in the paper's notation:

\mathcal{L}_{\text{cam}} = \sum_{i=0}^{n}\|\bm{q}_i - \hat{\bm{q}}_i\|_2 + \sum_{i=1}^{n}\Big\|\tfrac{\bm{T}_i}{\|\bm{T}_i\|_2} - \tfrac{\hat{\bm{T}}_i}{\|\hat{\bm{T}}_i\|_2}\Big\|_2 + \sum_{i=1}^{n}|p_i - p_{\text{gauge}}| + |\hat{p}_{\text{gauge}} - p_{\text{gauge}}| eq. 2

\bm{P}_i^{t\to t'} = \bm{P}_i^{t} + |t'-t|\,\bm{M}_{i,1}^{t}, \quad \bm{P}_i^{t-1\to t'} = \bm{P}_i^{t-1} + |t'-(t-1)|\,\bm{M}_{i,2}^{t-1} eq. 5

\mathcal{L}_{\text{retarget}} = \sum_{i=0}^{n}\Big(\|\hat{\bm{I}}_i^{t\to t-1} - \hat{\bm{I}}_i^{t-1}\|_2 + \lambda_{\text{S}}\,\text{SSIM}(\hat{\bm{I}}_i^{t\to t-1}, \hat{\bm{I}}_i^{t-1}) + \lambda_{\text{L}}\,\text{LPIPS}(\hat{\bm{I}}_i^{t\to t-1}, \hat{\bm{I}}_i^{t-1})\Big) eq. 6

\mathcal{L}_{\text{flow}} = \sum_{i=0}^{n} \bm{1}_{\text{cyc}}\,\|\bm{\mu}^{t\to t-1}_i - \hat{\bm{\mu}}^{t\to t-1}_i\|_2 eq. 7

\mathcal{G}_{3D}^{t'} = \{\bar{\mathcal{G}}_{3D}^{t\to t'}, \bar{\mathcal{G}}_{3D}^{t-1\to t'}, \bm{F}_{\theta}(\hat{\mathcal{G}}_{3D}^{t\to t'}, \hat{\mathcal{G}}_{3D}^{t-1\to t'})\} eq. 8

In equation 7, the cyclic weight 1_cyc is an exponential of the negative norm of the forward-backward flow mismatch, scaled by the hyperparameter r eq. 7. The mismatch uses the flow sampled at the warped pixel position eq. 7.

Hyperparameters

Values are from the paper unless marked "code". Where the paper says nothing, the code's value is given, marked as code-only.

Results

Author-reported throughout. The setup is 4 input views. Table 1 compares static novel-view synthesis at input timestamps with pose-free and posed baselines. Table 2 compares novel-time synthesis.

Table 1: static novel-view synthesis tab. 1

Table 2: novel-time and novel-view synthesis, sampling rate 2 tab. 2

In-the-wild, SelfCap and VVC, nine human-object scenarios, four cameras at 20° to 70° tab. 3:

Metric scale on the synthetic MetaHuman4D set, mean distance from predicted points to the ground-truth mesh tab. 4: MoGe-2 0.3309 m; Forge4D 0.0264 m. The text rounds the Forge4D figure to 0.02 m §4.2.

Dense motion on MetaHuman4D, motion error and retargeted point distance tab. 5: POMATO 0.01274 and 0.7555; Forge4D 0.00953 and 0.0215.

Speed on an NVIDIA H200 tab. 6 §4.3:

Ablation on components tab. 7:

Flicker, measured as temporal luminance standard deviation, is 0.5379 with fusion and 0.9446 without, which the paper reports as 75.6% higher §4.3.

Appendix results tab. 8 tab. 9 tab. 10 tab. 11. Camera count (PSNR / SSIM / LPIPS) tab. 8: 2 views 27.1316 / 0.9384 / 0.0648; 4 views 29.0378 / 0.9566 / 0.0535; 5 views 28.1337 / 0.9471 / 0.0613. Sampling rate tab. 9: 2 gives 29.0378 / 0.9566 / 0.0535; 4 gives 27.9129 / 0.9538 / 0.0604; 8 gives 25.9673 / 0.9344 / 0.0767. Video length tab. 10: 8 frames 29.0378 / 0.9566 / 0.0535; 16 frames 29.3932 / 0.9591 / 0.0519; 32 frames 29.3615 / 0.9591 / 0.0519. Speed against baselines on the H20 tab. 11: GPS-Gaussian 0.0703 s, 14.22 FPS; Queen 30.14 s, 0.725 FPS; LHM with Multi-HMR 5.330 s, 22.22 FPS; AnySplat 0.4636 s, 2.157 FPS; D-3DGS 8 min 38 s, 0.1453 FPS; SpaceTimeGS 1 h 52 min, 0.022 FPS; L4GM 0.2231 s, 71.72 FPS; Forge4D with 20 interpolations 0.5035 s, 39.72 FPS.

Paper vs code

Searched: the paper's abstract links a project page at zhenliuzju.github.io/huyingdong/Forge4D, and that page links github.com/zhenliuZJU/Forge4D. The project page sits under the first author's personal page, and the repository's README links the same page and repository. Cloned at commit 874308d93238d885feb9937cc40e36cde460d1bb. The repository ships code and configs only: the README lists all checkpoints as "coming soon" README.md.

Differences that the reader must know before implementing:

• Stage-3 training loop is unreachable as shipped. Trainer.train() runs one evaluation and then calls exit(), so the optimisation loop after it never runs train_matchedfusion.py. The stage 1 and 2 scripts have the same exit() commented out train.py train_dynamic.py.
• Motion head in stage 3. The paper says stage 3 trains the motion head with the fusion. The code's comments say the original release relied on a motion head from a separate training run, and that the motion head is now enabled inside the stage 3 branch train_matchedfusion.py. The repository's own audit file, PAPER_CONSISTENCY.md (sections A1 and B0), describes the same two-step original pipeline.
• Learning rates and step counts. The paper gives 5e-5 for every stage and 100k iterations per stage app. C. The shipped configs use 1e-6 for stages 1 and 3 and 5e-5 for stage 2, with 200k steps for stage 1 train_stage1_static.yaml train_stage1_static.yaml train_stage2_streaming.yaml train_stage3_motion_fusion.yaml.
• Flow loss. The paper's equation 7 uses an L2 norm and a cyclic weight with exponent r eq. 7 app. C. The code uses an L1 mean loss.py. It overwrites the occlusion mask with ones before using it loss.py. The weight comes from an uncertainty term whose threshold is 0.1 times the squared flow magnitude plus 0.5 flow.py. The loss is also divided by the image height train_matchedfusion.py. The shipped stage-3 config sets its weight to 10 train_stage3_motion_fusion.yaml. The paper gives no weight.
• Photometric terms. The paper's retargeting loss uses an L2 norm, a plain SSIM term and LPIPS with weights 0.25 and 0.25 eq. 6 app. C. The code uses 0.8 times the mean squared error plus 0.2 times one minus SSIM plus 0.2 times LPIPS train_matchedfusion.py loss.py.
• Gauge. The paper predicts a gauge with a head and uses it at inference §3.2 app. C. The code takes the mean of translation-length ratios during training, in the predicted-over-ground-truth direction, and rescales translations with it forge4d.py forge4d.py. The paper's ratio is written in the opposite grouping §3.2, so check the direction before implementing. Its scale_head outputs per-Gaussian scale channels, not a gauge dpt_head.py dpt_head.py. The stage 1 and 2 camera losses in the training scripts use per-camera translation-length ratios with a squared deviation from their mean, not the absolute deviation of equation 2 train.py.
• Zero initialisation is off. The paper says the heads and motion blocks are zero-initialised §4.1. The calls that would do this are commented out dpt_head.py dpt_head.py.
• State token count. The paper says "a learnable state token". The code defaults to 1024 tokens forge4d.py forge4d.py.
• Fusion distance. The paper does not say what distance the dual-consistency factor uses. The code divides the distance by a scale term before thresholding at 0.05 forge4d.py forge4d.py.

Pitfalls

• Ambiguous SSIM sign. The paper writes "+ λ SSIM(·,·)" in equations 6 and in the fusion loss, without saying whether it is a similarity or a dissimilarity eq. 6 §3.2. Adding raw SSIM would reward dissimilarity. The code uses one minus SSIM train_matchedfusion.py. Implement the dissimilarity form.
• Normaliser mismatch in the temporal gauge. The temporal gauge divides by (n−1)(k−1) but sums over all n·k ratios, with the sum starting at t = 1 §3.3. The paper does not explain this. Choose one convention and state it.
• Pose-free evaluation protocol. For NoPoSplat, AnySplat and Forge4D, the evaluation optimises the novel-view camera poses while keeping the Gaussians fixed app. C. Those numbers depend on this procedure, and it is not used at deployment app. C.
• Static comparison uses a different configuration. For the static comparison, the state tokens and state attention are disabled app. C, so Table 1 is not the same model as Table 2.
• Motion has no real ground truth. Dense motion accuracy is only measured on the synthetic MetaHuman4D set, with 11 identities and 48 render views app. B tab. 5.
• Constant velocity breaks at longer gaps. Novel-view PSNR falls to 27.91 at sampling rate 4 and to 25.97 at sampling rate 8 tab. 9. The conclusion names large motion and long gaps as limitations §5.
• Unstated losses in stage 2. The paper gives the temporal camera loss for stage 2 eq. 3 but not the photometric loss for that stage.
• Gauge is essential. Without gauge alignment the static model drops to 13.29 PSNR, and the paper says training collapses tab. 7 §4.3.
• Latency. The full model takes 224.27 ms per input frame pair, which gives 4.45 FPS. Interpolation adds about 1 ms per 10 steps, so output frame rate is high but input latency is not tab. 6.
• Hardware is inconsistent. Section 4.3 says the speed measurements are on an H200 §4.3, but the appendix comparison says H20 app. E.
• Metric figure is rounded. The text says 0.02 m, while the table reports 0.0264 m §4.2 tab. 4.
• Unexplained asterisk. Table 1 marks DualGS with 4D* and the paper does not explain it tab. 1.
• Fusion MLP details. The paper says the fusion MLP has two layers §3.4 and does not give its width or activation.
• Licences of vendored code. The repository is Apache-2.0 LICENSE and its README points to LICENSE for third-party terms. The vendored layers and the VGGT-based backbone were not audited for their own licences here. The stage-1 training also needs an external static VGGT checkpoint train_stage1_static.yaml.

Lineage

• Builds on: VGGT for the backbone and the camera and point-map heads §1 §3.2; 3D Gaussian Splatting for the representation §3.2; the linear motion model of FreeTimeGS §3.4; the polynomial 4D Gaussians of SpaceTimeGS, which motivate the constant-velocity approximation §3.4; SEA-RAFT for pseudo flow labels §3.4. No card exists for these yet.
• Compared against: DualGS, Queen and D-3DGS (optimisation); GPS-Gaussian (posed feed-forward); NoPoSplat and AnySplat (pose-free feed-forward); SpaceTimeGS and L4GM (4D); LHM with Multi-HMR; POMATO for motion; MoGe-2 for metric scale §4.1 §4.2 app. E.
• Superseded by: not known from content already in this wiki.