Learning Generalizable Feature Fields for Mobile Manipulation

In plain words

A robot that moves around a building and also picks things up needs one picture of the room that serves both jobs. For walking, it needs to know where the walls, people and furniture are. For grasping, it needs fine detail, such as the handle of a mug, and it needs to know what each part is called. Most existing systems keep these in separate maps, or build a detailed model of one room slowly, which does not work well in big spaces.

This paper proposes a learned "feature field" that keeps both. The network looks at a short stream of camera images with depth and their camera positions, and produces a compact set of latent points in one forward pass. Other small networks read that representation at any 3D position to get a distance to the nearest surface and a feature that can be matched against a text prompt. The network is first trained to predict views of new scenes, which teaches it the shape of rooms, and then it is trained to agree with the features of an image model that understands language.

The system runs on a four-legged robot with an arm and answers queries such as "the handle of the mug" with 0.39 s latency. The authors report higher overall success than two earlier open-vocabulary scene methods on their own robot tests.

TL;DR

• GeFF encodes posed RGB-D frames into sparse latent points with a generalizable NeRF encoder, with no per-scene optimisation §3.2.
• Two decoders read the latents: a signed distance field for geometry and a feature decoder distilled from MaskCLIP for language §3.2 eq. 2 eq. 4.
• Pre-trained on ScanNet for 50 epochs; the paper reports 6 days on 8 RTX 3090 GPUs §4.1.
• Author-reported real-robot results: 62.4% overall success vs 43.2% for LERF and 41.9% for ConceptGraphs (offline) tab. 1.
• Several training hyperparameters (learning rate, batch size, loss weights, feature width) are not stated in the paper, and no official code was found (see Paper vs code).

Problem

Navigation systems usually use geometric or topological maps and do not carry the detail needed for grasping. Manipulation systems often use dense surfaces that navigation does not need §1. Open-vocabulary methods that fuse 2D model outputs into explicit maps either need offline batch processing or store too much for room-scale scenes §2. LERF and feature-distillation NeRFs need costly per-scene optimisation, and point-based methods such as ConceptGraphs do not give part-level responses §2 §4.2 A1, A2, A3.

The method assumes the scene has been observed from several views with known poses (the paper uses an onboard VIO estimate §3.3) and that a 2D teacher model gives useful features for the target language §3.2.

Method

Overview. GeFF is a scene-level generalizable NeRF that is pre-trained on novel-view synthesis and then distilled from a 2D vision-language teacher §3.2 fig. 2. The encoder takes N posed RGB-D frames and outputs N latent vectors of width C. Decoders map any 3D point and the latents to geometric and semantic features §3.1. A single feed-forward pass updates the representation; no per-scene fitting is needed §1.

Requirements. The paper lists four design goals: the encoding is sufficient for both geometric and semantic queries; new observations can be added incrementally; the latents are sparse and implicit; and the semantics are open-set and aligned with language §3.1.

Step 1. Encode the frames §3.1 §3.2. The trainable encoder \hat f_{enc} consumes the RGB-D frames. The paper concatenates the teacher's 2D features of the same input views to the encoder output, so the latent is f_{enc}(\mathcal{D}) = \mathrm{ConCat}(\hat f_{enc}(\mathcal{D}), f_{teacher}(\mathcal{D})) eq. 3. The teacher is frozen §3.2.

Step 2. Geometry decoder. A signed distance network s(\mathbf{x}) = g_{geo}(\mathbf{x}, f_{enc}(\mathcal{D})) gives depth information. It is conditioned on the latents, so new scenes need no optimisation §3.2.

Step 3. Opacity from the SDF. Opacity along a ray is taken from the SDF with a sigmoid of learnable sharpness \sigma_s §3.2 eq. 4.

Step 4. Semantic decoder. A feature decoder g_{sem}(\mathbf{x}, f_{enc}(\mathcal{D})) maps a 3D point to a feature vector. It is isotropic, so it does not take the viewing direction §3.2.

Step 5. Render. Features and depth are volume-rendered along each novel-view ray with the transmittance and the SDF opacity §3.2 eq. 2 eq. 5. The paper says the rendering equations come from NeRF's colour rendering eq. 1.

Step 6. Losses. Features are trained with an L2 loss against the teacher's features on the ground-truth novel views §3.2. Depth is trained with an L2 loss on rendered depth §3.2 eq. 5. The SDF is trained with the iSDF SDF loss and an Eikonal regulariser §3.2. The total loss is a weighted sum of colour, depth, SDF, Eikonal and feature terms eq. 6. The paper says feature quality is not sensitive to the weights §3.2.

Step 7. Decode for the robot §3.3. The latent is decoded into a discretised point cloud. Points are projected to a 2D grid, and each cell averages the features of the points that fall into it. Text queries are compared with point or cell features by cosine similarity under a temperature softmax; positive-query probabilities are summed to give the score §3.3.

Step 8. Navigation §3.3. The robot is treated as a 2D navigation problem. DBSCAN clusters high-response cells into a goal, cells get semantic affordances, and an affordance-aware A* planner plans the path. The 2D occupancy map is updated in real time §3.3.

Step 9. Object grasp §3.3. The target object is found by its semantics, and DBSCAN gives its centroid. The gripper runs an open-push-close sequence, with the trajectory from an OMPL sample-based planner §3.3.

Step 10. Part grasp §3.3. For parts such as handles, the object centroid is found first. The in-wrist camera can add views for millimetre-scale detail. Conditional CLIP queries (from LERF-TOGO) segment the part, and DBSCAN with a much smaller EPS (about 1 cm) chooses the grasp point §3.3.

# GeFF pre-training step  [§3.2] [eq. 1-6]
# Input:  posed RGB-D input views and novel views with ground truth
# Output: scalar loss

def training_step(input_views, novel_views):
    # Encode the inputs; teacher features are concatenated to the latent  [eq. 3]
    teacher_inputs = teacher(input_views)                  # frozen teacher
    latents = concat(encoder(input_views), teacher_inputs)
    # Render feature and depth along each novel-view ray  [eq. 2] [eq. 5]
    for ray in rays_of(novel_views):
        # Opacity from the SDF with a learnable sharpness  [eq. 4]
        alpha = sdf_opacity(sdf_decoder(points, latents), sharpness)
        # Transmittance-weighted sums of semantic features and depth
        features = render(alpha, semantic_decoder(points, latents))
        depth = render(alpha, distances)
    # Weighted sum of the five losses  [eq. 6]
    return lambda_1 * l_color + lambda_2 * l_depth + lambda_3 * l_sdf \
        + lambda_4 * l_eikonal + lambda_5 * l_feature

The pseudocode is a condensed reading of the equations. The paper does not give a colour loss or colour decoder in the text it uses for GeFF, although the total loss in eq. 6 includes \mathcal{L}_{col}. See Pitfalls.

Hyperparameters

Results

Author-reported. Real-robot tests in four rooms: a 25 m² lab, a 30 m² meeting room, a 60 m² community kitchen and a 15 m² office §4.1. Object-level tests use 17 objects, 8 of them from categories not seen in pre-training §4.1. Part-level tests use 3 categories with 4 instances each §4.1. Trial counts per cell are not stated for the main table.

Main comparison tab. 1. Columns: navigation success and mobile manipulation success for object-level tasks; success for three part queries; the average of the part-level results; and the overall score, which is the mean of mobile manipulation and the part average.

The offline rows are marked in the paper as requiring batch processing of all observations tab. 1. The paper says the 19.2-point gain over the best baseline is on the overall score §1.

Scene change tab. 3: objects are added after the initial scan. Successes out of 9 trials.

Auxiliary teacher input ablation tab. 2. Object-level, meeting room and kitchen, out of 66 trials in total.

Geometry from compressed latents tab. 4, depth L2 error on 10 ScanNet validation scenes, units not stated. Both the GeFF and projection baselines are reduced to at most 512 points §4.2 A6.

Qualitative only: dynamic person detection, part manipulation, narrow doorway, lawn-avoidance navigation and door opening fig. 5 §4.2 A7.

Paper vs code

No official code.

The paper says the authors "plan to release the pre-trained models and the source code" §1. The search for a repository found nothing:

• the paper text and the arXiv metadata contain no repository link;
• the project page geff-b1.github.io lists the paper, the arXiv id and a Nerfies template credit, with no code or model link;
• GitHub repository searches for "GeFF generalizable feature fields", "geff feature field", and an author-name query each returned zero results.

Pitfalls

• Teacher identity is inconsistent. The method says MaskCLIP is the teacher §3.2, while the training details say "We use the ViT-L CLIP model as f_{teacher}" §4.1. An implementer has to choose; the paper does not say whether MaskCLIP is applied to a ViT-L CLIP backbone.
• Colour loss is in the objective but not defined. \mathcal{L}_{col} appears in eq. 6, but the text never gives a colour decoder or colour rendering for GeFF, which is described as isotropic in features and driven by SDF and depth §3.2.
• \Delta and sampling are undefined. The SDF opacity eq. 4 uses s(\mathbf{x} + \Delta), but the step size, the number of samples per ray, and the near and far bounds t_n, t_f are not stated.
• Opacity sharpness. \sigma_s has a learnable parameter s eq. 4, but its initialisation and any clamp are not stated. Division by \sigma_s(s(\mathbf{x})) can be numerically sensitive near the surface.
• Eq. 3 is recursive as written. f_{enc} on the left is defined from \hat f_{enc} and f_{teacher}, so the reader has to take it as the encoder output with teacher features appended.
• Feature targets differ by view. The feature loss compares rendered features with the teacher's features on ground-truth novel views, while the encoder sees the input views, which are adjacent but different §3.2.
• Coarse teacher features. The paper says MaskCLIP features are coarse §3.2, and part-level grasping optionally uses in-wrist views for finer detail §3.3.
• Small trial counts. Scene-change cells use 3 trials per object §4.2 A4, and table cells are out of 9 or 18. Confidence intervals are not reported.
• Baselines were run in different modes. ConceptGraph and LERF were run offline on a desktop with all frames batched, and ConceptGraph-Online on the robot with dropped frames §4.1. The overall comparison mixes online and offline timing.
• Part queries are analysed for ConceptGraphs only. The paper attributes ConceptGraphs' part failures to missing part data in its detector §4.2 A2. It gives no similar analysis for GeFF's conditional CLIP queries.

Lineage

• Builds on: generalizable NeRFs (pixelNeRF, GNT, and the RGB-D Gen-NeRF encoder of §2 §3.2); MaskCLIP for the teacher §3.2; iSDF for the SDF loss §3.2; the conditional CLIP query of LERF-TOGO §3.3.
• Compared against: LERF (per-scene feature field) and ConceptGraphs (point-based open-vocabulary scene graph) §4.1 tab. 1.
• Superseded by: not known from this wiki.