ARF: Artistic Radiance Fields

In plain words

Artists can paint a 2D picture in a style, but turning that look into a real 3D scene, one you can walk around, is hard. This paper takes a 3D model of a real place that already renders any viewpoint, and changes its colours and fine texture so that every view looks like a chosen style image, such as a painting.

Older methods built the 3D scene as a rough surface or a cloud of points, so mistakes in that geometry showed up in the result. This method works on the more robust 3D model itself. It also changes how the style is compared. The usual way of matching a painting's look averages it away, which blurs the brushstrokes. Here, each small region of a rendered view is matched to its closest counterpart in the painting, which keeps the strokes sharp. Because all views update one shared 3D model, the stylised views stay consistent as the camera moves. Users in the authors' study preferred these results over two earlier 3D-stylisation methods.

TL;DR

ARF starts from a photorealistic radiance field, keeps its shape fixed, and optimises only its appearance. The appearance is trained with a nearest-neighbour VGG feature loss, computed on full-resolution renders through a memory-saving deferred back-propagation, and with a linear colour transfer applied before and after. The paper reports user preference rates of 86.8% and 94.1% against two baselines, and no quantitative image metric.

Problem

Stylising a real 3D scene by editing a mesh or point cloud fails when the reconstructed geometry is wrong, which is common for complex real scenes §1 §2. Gram-matrix style losses, the usual choice for 2D style transfer, average out local detail and give blurry strokes in 3D §1 §4.1. The usual sparse per-pixel training of radiance fields cannot compute losses that need whole images, such as VGG-based style losses §1 §4.2. Each style needs its own optimisation, which takes minutes to tens of minutes per scene §5 (Limitations).

Method

The pipeline is an optimisation of an existing radiance field. The authors use Plenoxels as the field §4.4. They also apply the method to NeRF and TensoRF §4.4 fig. 8.

1. Start from a photorealistic field and freeze its geometry. Only the appearance part is optimised. The view-dependent colour is removed: all spherical-harmonic terms except the first are dropped, and for TensoRF the view directions are zeroed before the MLP §4.4. The radiance field is a function of position and view direction that is rendered by volume rendering §3 eq. 1.

   \sigma, \mathbf{c} = \mathrm{RadianceField}(\mathbf{x}, \mathbf{d}) eq. 1

2. Recolour the training views. A linear map in RGB is fitted so that the mean and covariance of the recoloured view pixels match those of the style image §4.3. The same map is applied to the field itself in the released code (see Paper vs code).

   \mathrm{E}[\mathbf{A}\mathbf{c}] = \mathrm{E}[\mathbf{s}], \quad \mathrm{Cov}[\mathbf{A}\mathbf{c}] = \mathrm{Cov}[\mathbf{s}] §4.3

   The closed-form solution uses eigen-decompositions of both covariances. The map is applied with the offset that moves the mean to the style mean § Appendix (Derivation of A).

   \mathbf{A}(\mathbf{c}-\mathrm{E}[\mathbf{c}]) + \mathrm{E}[\mathbf{s}] § Appendix (Derivation of A)

   \mathbf{A} = \mathbf{U}_s \boldsymbol{\Lambda}_s^{1/2} \mathbf{U}_s^T\, \mathbf{U}_c \boldsymbol{\Lambda}_c^{-1/2} \mathbf{U}_c^T § Appendix (Derivation of A)

3. Pre-fit the field to the recoloured views. The field is first trained with a per-ray MSE loss on the recoloured training images. This serves as the initialisation for step 4 §4.3 §4.4. The number of epochs is not stated in the paper. The released scripts use 2 for LLFF scenes and 1 for Tanks and Temples and custom scenes (Hyperparameters).

4. Stylise with the NNFM loss plus a content term. Each iteration renders one training view chosen at random §4.4. The VGG-16 features of the render, the style image and the content image are computed with the same extractor. The conv3 block is used because it keeps the style details best §4.4 fig. 9. The loss is the NNFM style term plus λ times a content term §4.1 eq. 4. Each render feature is compared with every style feature. The cosine distance is used for the match eq. 3, and the nearest style feature gives each term eq. 2. Stylisation runs for 10 epochs with the learning rate decayed exponentially from 1e-1 to 1e-2 §4.4.

   \ell_{\textrm{nnfm}}(\mathbf{F}_{\textrm{render}}, \mathbf{F}_{\textrm{style}}) = \frac{1}{N}\sum_{i,j}\min_{i',j'} D\big(\mathbf{F}_{\textrm{render}}(i,j), \mathbf{F}_{\textrm{style}}(i',j')\big) eq. 2

   D(\mathbf{v}_1, \mathbf{v}_2) = 1 - \frac{\mathbf{v}_1^T \mathbf{v}_2} {\sqrt{\mathbf{v}_1^T\mathbf{v}_1\,\mathbf{v}_2^T\mathbf{v}_2}} eq. 3

   \ell = \ell_{\textrm{nnfm}}(\mathbf{F}_{\textrm{render}}, \mathbf{F}_{\textrm{style}}) + \lambda\,\ell_2(\mathbf{F}_{\textrm{render}}, \mathbf{F}_{\textrm{content}}) eq. 4

   Here λ sets the stylisation strength: a larger λ keeps more content and a smaller λ gives stronger stylisation §4.1 (Controlling stylization strength). The paper gives λ = 0.001 for forward-facing captures and λ = 0.005 for 360° captures §4.4.

5. Deferred back-propagation for full-resolution losses. The full render is computed without building a gradient graph. The loss gradient with respect to each rendered pixel is cached. The render is then recomputed patch by patch with the graph enabled, and the cached gradient is pushed into the field's parameters and accumulated §4.2 fig. 3. The memory cost is that of a single patch, while the loss sees the full image.

   PseudocodeCopy

   # One stylisation step, deferred back-propagation  [§4.2] [fig. 3]
   # Input:  field, one training camera, style image, content image
   # Output: gradient accumulated on the field's parameters
   
   def stylise_step(field, camera, style, content):
       # Render the full view without a gradient graph  [§4.2]
       with no_grad():
           full_render = field.render(camera)           # (H, W, 3)
   
       # Loss gradient for each rendered pixel  [§4.2] [eq. 4]
       pixel_grad = grad_of_loss(full_render, style, content)  # (H, W, 3)
   
       # Re-render each patch with a graph and push the cached gradient  [§4.2]
       for patch in patches(camera):
           with enable_grad():
               patch_render = field.render(patch)       # (h, w, 3)
           patch_render.backward(pixel_grad[patch])
   
       # Update only the appearance parameters; density is fixed  [§4.4]
       field.optimiser_step()

6. Colour-match again after stylisation. The views are rendered at the training cameras, and the colour map is fitted between these renders and the style image. The same map is applied to the colour values of the field §4.3.

The NNFM term is computed per position on the VGG feature map, which is why it keeps local detail where a Gram matrix would average it §4.1. The paper describes the matching as nearest-neighbour search on the style image's feature map §4.1 eq. 2. The nearest-neighbour search is exhaustive over all style positions.

# NNFM style term on VGG conv3 features  [§4.1] [eq. 2] [eq. 3]
# Input:  render features (C, H, W), style features (C, Hs, Ws)
# Output: scalar style loss

def nnfm_style_loss(render_feats, style_feats):
    # One feature vector per position  [§4.1]
    queries = flatten_positions(render_feats)      # (C, N)
    keys = flatten_positions(style_feats)          # (C, M)

    # Cosine distance from each query to every key  [§4.1] [eq. 3]
    distances = 1 - normalise(queries).T @ normalise(keys)  # (N, M)

    # Nearest style feature for each render position  [§4.1] [eq. 2]
    nearest = distances.min(axis=1)                # (N,)
    return nearest.mean()                          # average over N positions

Hyperparameters

Values are from the paper where cited. Items marked "not stated" are absent from the paper. Items marked "code" appear in the official code only, and the paper does not describe them.

Results

Author-reported. The paper's evaluation is a user study and qualitative comparison, with no numeric image metric reported §5 (Qualitative comparisons) fig. 4 fig. 5.

Setup: four forward-facing scenes (Flower, Orchids, Horns, Trex) from the NeRF data, and seven 360° scenes (Family, Horse, Playground, Truck, M60, Train from Tanks and Temples, and Real Lego from the Plenoxels data) §5 (Datasets). Style images include a neon tiger, Van Gogh's The Starry Night, and sketches §5 (Datasets). Baselines are the released code and pre-trained models of Huang et al. (2021), a point-cloud method, and Chiang et al. (2022), a hypernetwork method §5 (Baselines).

The user study uses 5 batches of 5 questions. Each pair gets about 12 ratings on average §5 (User study). The paper reports no significance test.

Paper vs code

The official code is at https://github.com/Kai-46/ARF-svox2, at commit 22baf6e000e519b21ca26308f8ead1f4d2675cf5. The repository has a BSD-2-Clause license, which covers the ARF and Plenoxels authors. The owner is the first author. The project page links it as "Code & Data". A GitHub repository search for "ARF artistic radiance fields" returned the same repository.

Checks that agree with the paper:

• λ values match the scripts: 1e-3 for LLFF (opt/try_llff.sh#L21) and 5e-3 for Tanks and Temples and custom scenes (opt/try_tnt.sh#L22, opt/try_custom.sh#L21) §4.4 eq. 4.
• VGG conv3 block is the default [code: opt/opt_style.py#L51] §4.4.
• 10 NNFM epochs [code: opt/opt_style.py#L69] §4.4.
• Density fixed by lr_sigma 0 [code: opt/configs/llff_fixgeom.json#L4-L5] §4.4.
• Colour-transfer formula matches the appendix [code: opt/nnfm_loss.py#L6-L39] § Appendix (Derivation of A).
• Learning rate endpoints 1e-1 and 1e-2 match in the fixgeom configs [code: opt/configs/llff_fixgeom.json#L6-L7] §4.4.

Differences, each with both citations:

• Image total-variation term. The paper's loss has no such term eq. 4. The code adds one with weight 1 [code: opt/opt_style.py#L47, opt/opt_style.py#L611-L613].
• Content term. The paper writes an ℓ2 difference eq. 4. The code uses a mean squared error over all elements, not a sum or norm [code: opt/nnfm_loss.py#L181-L183]. The value of λ therefore means something different in each.
• Downscaling before VGG. The paper does not mention a resolution change. The code halves render, style and content before the VGG features [code: opt/opt_style.py#L615-L631]. The style image is also resized to the training images' long side, then halved [code: opt/opt_style.py#L507-L527].
• Colour transfer on the field. The paper says the training views are recoloured for the pre-fit §4.3. The code also applies the same map to the field's colour coefficients before the pre-fit [code: opt/opt_style.py#L538-L540, svox2/svox2.py#L1567-L1576].
• Clamping and stabilisers. The colour map clamps eigenvalues to [1e-8, 1e8] and clamps output to [0, 1] [code: opt/nnfm_loss.py#L27-L28, opt/nnfm_loss.py#L34]. Neither appears in the paper.
• Optimiser. The paper gives a learning-rate range but no optimiser. The code's default is RMSprop [code: opt/opt_style.py#L178, #L255].
• Pre-fit epochs. The paper does not state them. Scripts use 2 for LLFF and 1 for other sets [code: opt/try_llff.sh#L20, opt/try_tnt.sh#L21].
• Background model and shared-SH total variation. The paper describes neither. The Tanks and Temples and custom configs use a 64-layer background model [code: opt/configs/tnt_fixgeom.json#L4, opt/configs/custom_fixgeom.json#L4]. The fixgeom configs also set lambda_tv_sh to 5e-3 [code: opt/configs/llff_fixgeom.json#L11, opt/configs/tnt_fixgeom.json#L19].
• Decay horizon. The paper gives the endpoints only. The code's default horizon is 250000 steps [code: opt/opt_style.py#L181]. The step counter includes the pre-fit ray batches [code: opt/opt_style.py#L768]. A short run may therefore barely decay the rate.
• Radiance field in the code. The official pipeline trains a Plenoxels (svox2) field first, if no checkpoint exists [code: opt/try_llff.sh#L11-L14, README]. The paper does not describe that field's training.
• Code licence. The LICENSE file is BSD-2-Clause and names the ARF and Plenoxels authors [code: LICENSE]. This applies to the repository, and I did not quote any code in this card.

Pitfalls

• Recipe order matters. Colour matching appears before and after the stylisation step in both the paper and the code. The code also transforms the field before the pre-fit. An implementation that only recolours the images gets a different starting point §4.3 [code: opt/opt_style.py#L538-L540].
• The paper's loss alone does not reproduce the code's loss. The image TV term and the 0.5 resize change the loss scale. The scripts' λ and learning rates were run with the code's version, so tune against that version §4.1 [code: opt/opt_style.py#L615-L631].
• The learning rate is an RMSprop step size in the code. Adam or SGD with the same numbers will behave differently. The decay horizon is also unstated in the paper §4.4 [code: opt/opt_style.py#L178, #L181].
• The NN search is exhaustive and chunked. The distance matrix covers every render position against every style position. The code splits it into chunks of about 1e8 entries, so cost grows with style resolution [code: opt/nnfm_loss.py#L55-L63]. Time overheads of the deferred pass are not reported in the paper.
• Deferred back-propagation renders twice. The full render without a graph is followed by patch re-renders with a graph §4.2. The paper states the memory footprint of one patch but gives no time figure.
• Density is frozen. Floaters in the photorealistic field stay in the stylised result. The paper names this as a limitation §5 (Limitations).
• Each style needs a full optimisation. The paper reports about 3 minutes for forward-facing scenes and about 20 minutes for 360° scenes on one RTX 3090 §5 (Limitations). The stylised field renders in real time once trained.
• No editing. The paper states that the artistic field cannot be edited by hand §5 (Limitations).
• Results are author-reported. The only numeric result is the user study, with about 12 ratings per pair and no significance test §5 (User study).

Lineage

• Builds on: Plenoxels, NeRF and TensoRF as radiance fields §1 §4.4; Gatys et al. (2016) neural style transfer with a Gram loss, which the paper replaces §2 §4.1; CNNMRF and NNST nearest-neighbour style losses. The paper's NNFM loss is closest to NNST, but without coarse-to-fine stylisation and without rotation or scale augmentation of the style image §2.
• Compared against: Huang et al. (2021), featurised point clouds with a 2D CNN renderer §2 §5 (Baselines); Chiang et al. (2022), a hypernetwork over neural radiance fields §2 §5 (Baselines).
• Superseded by: not stated in this card's sources.