Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

In plain words

Give the system one photo of a scene, such as a coastline, plus a camera path. It produces a video that keeps flying further into the scene, hundreds of frames long. Older view-synthesis methods only handle small camera moves, and video generators make only a few dozen frames, so both lose the scene quickly.

The trick is to repeat three steps. First, draw the current picture from the new camera position, using a depth-like map (disparity is inverse depth) so that pixels move in a geometrically correct way. Second, refine the drawing: fill the gaps and add detail. Third, feed both the picture and its depth map into the next round. Carrying the depth map forward from frame to frame is what keeps the scene from drifting.

The model is trained on aerial footage of coastlines. The authors also wrote a simple automatic camera pilot that steers away from the ground and the sky. The result is long, plausible flights from one image, but nothing keeps the invented world consistent over time.

TL;DR

• Single-image, long-horizon view generation with a render, refine, repeat loop that carries both RGB and disparity between steps.
• Trained with 5 unrolled steps on the authors' ACID dataset of aerial coastline video; generates 500 frames at test time §6.3.
• Author-reported: FID-50 of 50.6 over 50 frames, against 65.0 for the best baseline tab. 1.
• Code is Apache-2.0 and covers inference only. The training loop, optimizer and schedule are not in the repository, so training details rest on the paper.

Problem

• View synthesis methods that work from one image fail once the camera moves outside a limited range §1 fig. 6.
• Video generators are limited in length: the two cited methods make 25 and 48 frames, and they largely ignore scene geometry §1.
• Methods that build one static 3D representation of the scene are limited by its extent. Some augmentations extend that range, but only a little §2.
• Assumptions: the scene is mostly static, the camera moves, and the training videos are real aerial footage §2. Dynamic objects are not modelled §7.

Method

The model is a loop with three parts, applied once per camera step §3 §3.1 fig. 2 fig. 3. Given the current RGB image I_t, its disparity D_t, and the current and next camera poses P_t and P_{t+1}:

1. Render. A differentiable mesh renderer lifts each pixel into 3D using its disparity and the camera intrinsics, then draws the RGB and disparity from the new pose §3.1 eq. 1. The rendered mask marks the holes, which come from disocclusions or leaving the field of view §3.1.
2. Mask. A pixel is treated as a hole where the Sobel gradient magnitude of the input disparity is above a threshold \alpha §3.1 eq. 3. This stops stretched triangles at depth edges. The masked render is the input to refinement.
3. Refine. A generator fills holes, adds detail and super-resolves. The paper treats this as image-to-image translation with a SPADE generator, not as compositing, because super-resolution is also needed §3.1. Its input is 5 channels: RGB, disparity and the mask app. A.4. Its noise comes from a variational encoder applied to the initial RGB-D frame I_0 §3.1 app. A.4. Output: a new RGB image and disparity eq. 2.
4. Re-ground. The refined disparity is rescaled by one factor \gamma, fitted so that the refined disparity matches the rendered disparity on the visible pixels §3.1 eq. 4. This keeps the disparity from drifting over long runs.
5. Repeat. The refined RGB and disparity become the input to the next step §3.1.

# Perpetual view generation loop  [§3] [§3.1]
# Input:  image (H, W, 3), disparity (H, W), poses (P_0 ... P_T)
# Output: frames and disparities, one per pose

def generate_sequence(image, disparity, poses):
    # Noise from the initial RGB-D frame, its mean at test  [app. A.4]
    style = encoder_mean(stack(image, disparity))
    frames = [image]
    disparities = [disparity]
    for pose, next_pose in zip(poses[:-1], poses[1:]):
        # Warp RGB-D to the next camera; mark holes  [eq. 1] [eq. 3]
        rendered, mask = render(image, disparity, pose, next_pose)
        # Fill holes, add detail, super-resolve  [§3.1] [eq. 2]
        refined = refinement(rendered, mask, style)
        # Rescale refined disparity on visible pixels  [eq. 4]
        disparity = rescale(refined[..., 3], rendered[..., 3], mask)
        image = refined[..., :3]
        frames.append(image)
        disparities.append(disparity)
    return frames, disparities

Equations (reproduced in our notation; the paper's numbering is kept)

(\hat I_{t+1}, \hat D_{t+1}, \hat M_{t+1}) = \mathcal{R}(I_t, D_t, P_t, P_{t+1}) eq. 1

(I_{t+1}, D_{t+1}) = g_\theta(\hat I_{t+1}, \hat D_{t+1}, \hat M_{t+1}) eq. 2

M_t = \begin{cases} 0 & \text{where } \|\nabla \hat D_t\| > \alpha \\ 1 & \text{otherwise} \end{cases} eq. 3

\gamma = \arg\min_{\gamma} \| M \odot (\log(\gamma D_{t+1}) - \log \hat D_{t+1}) \| eq. 4

Setting the derivative to zero gives \log\gamma = \sum M (\log \hat D - \log D) / \sum M (derived, least squares). The code computes this form infinite_nature_lib.py.

Training. The model is trained on sequences of frames from ACID, rolled out for T = 5 steps of the loop above, with losses backpropagated through every step §3.1 §5. The loss has five terms: an L1 term over colour and disparity, a VGG-based perceptual term on colour, a hinge adversarial term with feature matching from a discriminator, and a KL term on the initial encoder §5. The loss is computed over all steps and all batch samples §5. The paper's total is:

\mathcal{L} = \mathcal{L}_{\text{reconst}} + \mathcal{L}_{\text{perceptual}} + \mathcal{L}_{\text{adv}} + \mathcal{L}_{\text{feat matching}} + \mathcal{L}_{\text{KLD}} eq. 5

Its weights are given only in the appendix: reconstruction 2, perceptual 0.01, adversarial 1, KLD 0.05, feature matching 10 app. A.5. The loss is written in pseudocode below with those weights.

# Training loss over all steps and batch samples  [§5] [app. A.5]
# Weights from the paper's appendix; the code file differs (see below)
def training_loss(fake, real, disc, mu, logvar):
    # L1 reconstruction on RGB and disparity  [§5]
    reconst = mean(abs(fake - real))
    # VGG perceptual loss on RGB; layers not stated  [§5]
    perceptual = vgg_loss(fake[..., :3], real[..., :3])
    # Hinge adversarial and feature matching losses  [§5]
    adversarial = hinge_loss(disc(fake))
    feat_match = feature_match(disc(fake), disc(real))
    # KL of the initial encoder to a standard normal  [§5]
    kld = kl_to_standard_normal(mu, logvar)
    return (2.0 * reconst + 0.01 * perceptual
            + 1.0 * adversarial + 10.0 * feat_match
            + 0.05 * kld)

Training resources. Batch size 4 on 10 GPUs, 7 million iterations, about 8 days; the checkpoint is chosen by best FID on a validation set §5. The number of steps T is limited by memory and the available sequence lengths §5. The discriminator sees random crops and random generated frames to save memory app. A.4.

Training data sub-sampling. Frame rates are chosen so that camera speed falls in a target range that varies by up to 30%; on average 90% of the image stays visible in the next sampled frame app. A.3.

Test-time camera control. The trajectory can be given, generated by an auto-pilot, or steered by a user §3 §6.4. The auto-pilot works as follows app. A.7:

• Each frame, classify pixels as sky if disparity is below 0.05 and as near if above 0.5.
• Look up or down so that 30% of the image is sky.
• Look toward the side with more sky.
• Move up when the near share exceeds 20%, down when it falls short; otherwise head toward a centre point 30% from the top.
• Smooth the look and move directions by moving 5% of the way to the target each frame, then advance a fixed distance along the move direction.

Data pipeline (ACID). Videos were collected with place names and drone keywords app. A.1. Each video goes through a SLAM and structure-from-motion pipeline modelled on the one of Zhou et al. §4 app. A.1. The authors remove static, non-aerial and people-heavy videos, drop frames where people cover more than 10% of the image, and crop letterbox borders while adjusting intrinsics app. A.1. MiDaS disparity is fitted to the sparse SfM points with a per-frame scale and shift by least squares §4 app. A.1 eq. 6. At test time the raw MiDaS output can be used without this scaling, with a negligible effect on FID app. A.2 fig. 10.

\min_{a,b} \sum_{(x,y,z) \in \mathcal{K}} \big(a \hat D_{xyz} + b - z^{-1}\big)^2 eq. 6

Frame interpolation (supplementary video only). Extra poses between two generated frames are rendered from both frames and blended app. A.8 eq. 7. The paper states that all reported numbers use raw outputs without interpolation app. A.8.

Hyperparameters

Results

Author-reported. Setup: on ACID test sequences, the input frame is followed by ten ground-truth frames, and the camera moves forward at up to 45° §6.1. The trajectory is extended to 50 steps by extrapolation, and sequences that would collide with the scene are removed, leaving 279 §6.1. LPIPS and MSE are measured on the first ten frames; FID-50 on all 50 §6.1. The baselines are SynSin, single-view MPI, 3D Photos and SVG-LP, each retrained on ACID except 3D Photos §6. Their iterated and repeat variants are also compared §6 tab. 1.

The paper states that its LPIPS and MSE are competitive with the view-synthesis baselines, and that its FID-50 is much lower than the others, reflecting more natural output §6.1. The best LPIPS in the table is 3D Photos at 0.30 tab. 1.

Degradation over time: a sliding FID-20 from frame 10 to 40 shows the baselines getting worse faster than the method §6.1 fig. 5.

Long generation: FID-50 over 500 generated frames is stated only in a plot. The paper says it is below the baselines' FID-50 over 50 frames. No number is given in the text §6.3 fig. 5 fig. 8.

Ablations tab. 2. The protocol is not restated in §6.2 beyond the table caption.

With T=1, LPIPS and MSE stay similar to the full model, but FID-50 rises from 50.6 to 95.4 §6.2 tab. 2. Omitting the refine step leaves a growing share of the image empty §6.2 fig. 6. A U-Net instead of the SPADE generator is much worse §6.2 app. A.4. Disabling re-grounding costs a little FID §6.2 fig. 14.

Paper vs code

Checked against the official repository google-research/google-research at infinite_nature/ (commit d3a4bd8f0eceddda553e47085033b537935b283f). The project page (https://infinite-nature.github.io/) links it. The folder has no LICENSE of its own; the repository root LICENSE is Apache-2.0. The README says the code is not an officially supported Google product README.md.

• Training is not in the repository. There is no training loop, optimizer, learning rate, batch size or schedule. The loss file is a "reference implementation" that the demo does not use loss_impl.py. Training values come only from the paper §5.
• Loss weights differ. The paper gives reconstruction 2 and adversarial 1 app. A.5. The reference file uses 0.5 and 1.5, and scales the discriminator loss by 1.5 loss_impl.py. The file calls its lambdas the "default values used during the original submission" loss_impl.py. We keep the paper's values and flag the conflict.
• Perceptual loss. The paper names VGG and gives no layers §5. The reference file uses VGG-19 from conv1_2 to conv5_2, with divisors 2.6, 4.8, 3.7, 5.6 and 10/1.5 on conv5_2 loss_impl.py. Its VGG builder raises NotImplementedError loss_impl.py. The call passes the 4-channel RGBD tensor, while the docstring asks for RGB loss_impl.py loss_impl.py.
• KL term. The paper gives the KL to \mathcal{N}(0,1) §5. The code sums over all latent elements instead of averaging loss_impl.py.
• Latent size. The reference file's docstring says 128 loss_impl.py. The encoder and config use 256 networks.py config.py. The paper does not state it.
• Discriminator. The paper cites Pix2PixHD with two scales app. A.4. The reference file has four layers, 64 base channels, instance norm and two scales loss_impl.py.
• Re-grounding details. The code applies re-grounding at each inference step infinite_nature_lib.py. It adds clipping (input to [0.01, 1], output to [0, 1]) and a 1e-4 term in the denominator infinite_nature_lib.py. The paper states none of these. The training-time use is not in the code.
• Mask threshold. The paper leaves α open eq. 3. The code computes \exp(-10 \cdot \text{Sobel magnitude}) > 0.3, which is the same as a Sobel magnitude below \ln(1/0.3)/10 \approx 0.12 render.py render.py. The Sobel magnitude uses a 1e-6 offset under the square root render.py.
• Auto-pilot. The paper gives hard thresholds of 0.05 and 0.5 and a three-way move rule app. A.7. The code uses soft ramps: sky weight \text{clip}(20(0.1-d)) fly_camera.py, near weight \text{clip}(10(d-0.5)) fly_camera.py, and a continuous move height h = \text{horizon} + (\text{near} - \text{ground}) fly_camera.py fly_camera.py. Its direction matches the paper's rule, but it is not the same rule.
• Meander. The paper's turning experiment uses a 50-frame sinusoidal offset app. C.4. The demo's default magnitude is 0.0 autocruise.py.
• Field of view. The paper's smartphone example assumes 90° app. A.2 fig. 11. The demo and notebook assume focal 0.8, about 64° autocruise.py infinite_nature_demo.ipynb.
• Run length. The paper's long run is 500 frames §6.3. The default is 500 autocruise.py, but the README example uses 100 README.md.
• Checkpoint. The paper says about 7 million iterations §5. The released checkpoint is named for step 6935893 autocruise.py.
• Refinement width. The paper does not state it. The code hard-codes 32 networks.py. The config constant REFINEMENT_CHANNEL_SIZE (32) is not used in networks.py config.py.
• Style noise. The paper samples it during training and uses the mean at test app. A.4. The demo computes it once per sequence from the initial frame and reuses it at every step infinite_nature_lib.py autocruise.py. This is consistent with the test-time rule.
• Dataset counts. §1 says over 700 videos and 2 million frames §1. §4 says 891 videos, more than 13,000 sequences and 2.1 million frames §4.
• Consistent items. Resolution 160 × 256 §5 infinite_nature_lib.py. Five refinement inputs app. A.4 networks.py. Interpolation fraction 0.05 app. A.7 fly_camera.py. Hinge adversarial loss and KL term §5 loss_impl.py.

Pitfalls

• Drift is the main risk. The loop is memoryless and each step feeds on the last. The disparity drifts to very small values after about 150 frames without re-grounding, and the RGB then drifts too fig. 15 app. C.3. Re-grounding is needed for long runs §6.2 tab. 2 fig. 14.
• No global consistency. Content leaving the view is re-hallucinated on the way back. In a palindromic trajectory, the return frame does not match the input app. C.5 fig. 17. The authors state that long-term memory is open §7 app. C.1.
• Renderer returns the squared mask. render returns mask * mask render.py. The paper does not say so. For binary masks this changes nothing, but for fractional edge values it does.
• Mask threshold is sensitive. The cut-off is about 0.12 on the unnormalised Sobel magnitude, and it depends on the disparity scale render.py. Changing the scale of disparity changes the hole shapes.
• Disparity scale. Training uses MiDaS fitted to SfM points app. A.1. Test-time use of raw MiDaS has a negligible FID effect app. A.2 fig. 10. The demo expects disparity in [0, 1] autocruise.py, and the rescale step clips at 0.01 and 1 infinite_nature_lib.py.
• Resolution is fixed. The placeholders are [160, 256, 4] infinite_nature_lib.py, and H and W must be divisible by 32 networks.py. Other sizes need a code change.
• Intrinsics use unusual units. The code passes [0.8, 0.8 \cdot \text{aspect}, 0.5, 0.5] autocruise.py. Focal 0.8 gives about 64°. This is not pixels, and the aspect ratio of the input is used autocruise.py.
• Auto-pilot is a heuristic. It is not learned, and the authors call a learned version future work app. A.7. The step distance and the speed units are not stated app. A.7.
• Compute. The demo runs at "a few frames per second" on a free Colab GPU §6.4 fig. 9. Training took about 8 days on 10 GPUs §5. GPU type is not stated.
• Baseline fairness. The SynSin repeat variant ran with a reduced batch size, and the authors say so app. B.1. The MPI baseline blurs on repeated application §6.1. The repeat variants therefore do not share the full model's training setup.
• FID caveats. FID-50 uses 50 frames and reference statistics from 20k ACID images §5. Values are comparable only within one table tab. 1.
• Repeat matters for FID. With T=1, LPIPS and MSE are close to the full model (LPIPS slightly better, MSE slightly worse), but FID-50 rises from 50.6 to 95.4 tab. 2 §6.2.
• Dataset rights. The paper does not state a license for ACID. Its qualitative results use Pexels images, because ACID cannot be shown for licence reasons app. A.6. We use no dataset content here.

Lineage

• Builds on: SPADE as the refinement generator §3.1 app. A.4 (no card on this wiki); a differentiable mesh renderer §3.1; MiDaS for initial disparity §4; the SLAM and structure-from-motion pipeline of Zhou et al. §4.
• Compared against: 3D Photos, SynSin, single-view MPI and SVG-LP, with their iterated and repeat variants §6 tab. 1.
• Superseded by: not stated.