SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images

In plain words

Teaching a robot to pick up a named object usually needs a detailed 3D model of the table. Building that model normally takes many photos taken from many angles, and it has to be rebuilt whenever something moves. That is slow, and an arm cannot always reach the angles that would be needed.

SparseGrasp works from only a handful of photos. It builds the 3D scene out of many small soft blobs, and gives each blob a short label that says what it looks like in terms of words. A text instruction can then find the matching blobs. When someone moves an object, the system only moves that object's blobs until they match the new view, instead of rebuilding the whole scene. A grasp model then works directly on these blobs to propose where the gripper should close.

TL;DR

SparseGrasp is an open-vocabulary grasping pipeline for changing scenes. It fits a 3D Gaussian Splatting scene to three RGB views, initialised from DUSt3R dense points, with 16-dimensional PCA-compressed semantic features distilled from MaskCLIP and mask averages. Moved objects are re-posed by render-and-compare in about 200 ms. The paper reports 55 of 70 successful grasps on a real arm with three views, against 38 of 70 for F3RM with 17 views (author-reported).

Problem

Language-guided grasping methods that distil 2D features into NeRF or 3DGS scenes need dense views and long reconstruction. With only three views, F3RM is reported to struggle with object geometry, and even with 17 views its semantic distillation stays suboptimal §1 fig. 1. Capturing dense views also needs precise arm path planning, because some viewpoints are hard for the arm to reach §1. Reconstruction takes about 10 minutes for F3RM and 34 minutes for LERF-TOGO in the paper's setup tab. 4. Scene changes force a full re-capture and rebuild §1 §4.2.

The paper's assumptions:

• Only RGB images are used. The paper argues that RGB-D sensors add hardware requirements §3.1.
• The camera poses are calibrated and given to DUSt3R §4.
• "Changeable" means objects may be moved, and "multi-turn" means the robot runs a sequence of language commands §1.
• Adding or removing objects is not handled by the update; the scene must be re-initialised §4.3.

Method

SparseGrasp has four parts. Dense initialisation and semantic features come first §3.1. Then the features are compressed and distilled into 3DGS §3.2. The third part updates moved objects by render-and-compare §3.3. The last part grasps from the Gaussians §3.4. fig. 2 shows the pipeline.

1. Dense initialisation. DUSt3R, given the calibrated poses, produces a dense point cloud that is resampled to about 100,000 points and used to initialise the Gaussians §3.1 §4. The paper argues that sparse COLMAP points overfit with few views §3.1 fig. 3.
2. Dense semantic features per view. Patch-level features from MaskCLIP are upsampled to image resolution by nearest-neighbour interpolation §3.1. Masks from a segmenter are sorted from smallest to largest, and each pixel is assigned to the first mask that covers it, so smaller regions win overlaps §3.1. The features are averaged inside each mask, which gives a matrix F^sem with N mask rows and C channels §3.1.
3. Compression. PCA reduces the averaged features to 16 dimensions, giving F^sem_pca §3.2 §1. The 768 channels before PCA are stated in §1. fig. 9 shows other component counts §4.3.
4. Distillation into 3DGS. Each Gaussian carries a feature f_i. Feature maps are rendered with the same alpha-blending as colour eq. 1 eq. 2. The colour loss and a semantic loss are summed eq. 3 eq. 4 eq. 5. Unlike LangSplat, the opacity gradient comes only from the RGB loss §3.2.
5. Render-and-compare for changes. Moved pixels are found with MOG2 background subtraction between an initial frame and the current frame §3.3. The Gaussians whose features match the moved pixels are selected by cosine similarity. Only their translation and rotation are optimised, against the current views and a location term eq. 6 §3.3.
6. Grasping. GraspNet is retrained to take each Gaussian's centre, scale and rotation as input, instead of a point cloud of centres §3.4 fig. 4. Training uses 100 scenes from the GraspNet dataset. Object Gaussians are labelled "objectness" by reconstructing object-only and background-only versions and overlaying them. Random extra Gaussians and several densification thresholds add robustness §3.4.

Key equations, as written in the paper:

C = \sum_{i \in \mathcal{N}} T_i\, c_i\, \alpha_i, \qquad T_i = \prod_{j=1}^{i-1}(1-\alpha_j) eq. 1

\hat{F}^{sem}_{i} = \sum_{k \in \mathcal{N}} f_k\, \alpha_k\, T_k eq. 2 (the paper reuses index i for both pixel and Gaussian; k is used here for the Gaussian)

\mathcal{L} = \mathcal{L}_{rec} + \lambda_2\, \mathcal{L}_{sem} eq. 3

\mathcal{L}_{rec} = (1-\lambda_1)\, \mathcal{L}_1 + \lambda_1\, \mathcal{L}_{\mathrm{D\text{-}SSIM}} eq. 4

\mathcal{L}_{sem} = \mathcal{L}_1\big(\mathbf{F}^{sem}_{pca}, \hat{\mathbf{F}}^{sem}\big) eq. 5

\mathcal{L} = \mathcal{L}_1(I_{mov}, I_{pred}) + \lambda_3\, \mathcal{L}_1(d_{pred}, d_{gt}) eq. 6

The pseudocode below follows the paper's text. Steps the paper does not spell out are marked "not stated" in the comments. Weights are lam1 = 0.2, lam2 = 1, lam3 = 0.1 §3.2 §3.3.

# Build the scene from sparse views  [§3.1] [§3.2] [§4]
# Input:  images (N RGB views), calibrated camera poses (N)
# Output: 3DGS scene with one semantic feature per Gaussian

def build_scene(images, poses):
    # Dense initial points from DUSt3R, given the poses  [§3.1] [§4]
    points = dust3r_dense_points(images, poses)
    # Resample the cloud to about 100,000 points  [§4]
    points = resample(points, 100_000)
    # Patch-level CLIP features at image resolution  [§3.1]
    clip_maps = [upsample(maskclip_patches(img)) for img in images]
    # Masks per view, smallest first; earlier masks win overlaps  [§3.1]
    masks = [order_by_area(segment(img)) for img in images]
    # One averaged feature per mask: F_sem, shape (N_masks, C)  [§3.1]
    region_feats = mask_average(clip_maps, masks)
    # Compress to 16 dimensions with PCA  [§3.2] [§1]
    # Fit scope (per scene or global) not stated
    compressed = pca_fit_transform(region_feats, dims=16)
    # Initialise one Gaussian per point, each with a feature  [§3.2]
    scene = init_gaussians(points, feature_dim=16)
    for iteration in range(7000):  # 7,000 iterations  [§4]
        # Render colour and features with one rasteriser  [§3.2] [eq. 1] [eq. 2]
        rgb, feats = render(scene)
        # Colour loss with lam1 = 0.2  [§3.2] [eq. 4]
        loss_rec = (1 - lam1) * l1(rgb, images) + lam1 * dssim(rgb, images)
        # Semantic loss to the PCA targets, lam2 = 1  [§3.2] [eq. 5]
        # Mapping mask features back to pixels is not stated
        loss_sem = l1(feats, pixel_targets(compressed, masks))
        # Total loss  [§3.2] [eq. 3]
        loss = loss_rec + lam2 * loss_sem
        # Alpha gradient comes from the RGB loss only  [§3.2]
        scene = optimise(scene, loss)
        # Densify every 200 iterations  [§4]
        if iteration % 200 == 0:
            scene = densify(scene)
    return scene

# Update the scene after an object moves  [§3.3] [eq. 6]
# Input:  scene, initial frame, current views (3 in the experiments)
# Output: translation and rotation of the moved Gaussians

def update_moved_object(scene, frame_init, frames_now):
    # Changed pixels between the two frames, via MOG2  [§3.3]
    moved = mog2_changed_pixels(frame_init, frames_now)
    # Target: centre of the moved pixels in the current frame  [§3.3]
    d_gt = centre(moved)
    # Mean semantic feature of the moved pixels  [§3.3]
    # Frame used for this mean is not stated in the text
    query_feat = mean(scene_features_at(moved))
    # Gaussians closest to the query by cosine similarity  [§3.3]
    # Cosine threshold not stated
    moving = select_by_cosine(scene, query_feat)
    # Only translation and rotation are optimised  [§3.3]
    translation = zeros(3)
    rotation = identity_rotation()
    # Number of steps and optimiser not stated
    for step in range(num_steps):
        # Render the moving subset alone  [§3.3]
        image_obj = render(moving, translation, rotation)
        # Render the full scene with the subset moved  [§3.3]
        image_pred = render(scene, moving, translation, rotation)
        # Predicted object location on the subset render  [§3.3]
        # How this location is computed is not stated
        d_pred = locate(image_obj)
        # Pixel loss plus location loss, lam3 = 0.1  [§3.3] [eq. 6]
        loss = l1(frames_now, image_pred) + lam3 * l1(d_pred, d_gt)
        translation, rotation = step_optimiser(translation, rotation, loss)
    return translation, rotation

# Language-guided grasp from the 3DGS scene  [§3.4] [fig. 2]
# Input:  scene, instruction (text)
# Output: grasp candidates on the named object

def grasp_target(scene, instruction):
    # Find the Gaussians that match the text query  [fig. 2]
    # Text-to-feature mapping, including the PCA step, is not stated
    target = match_text(scene, instruction)
    # Retrained GraspNet reads centre, scale and rotation per Gaussian  [§3.4]
    candidates = grasp_net(scene.p, scene.s, scene.q)
    # Keep the candidates on the matched object  [§3.4]
    # Selection rule is not stated
    return keep_on(candidates, target)

Hyperparameters

Results

Author-reported. Setup: a real KINOVA Gen2 arm with three RGB cameras, on a desktop GPU §4. Static grasping uses seven objects with 10 attempts each, in a 70-trial total tab. 1. Baselines are F3RM and LERF-TOGO, each run with 3 or 17 views tab. 1. Scene-update tests use five scenes with five trials each tab. 2. Totals and the 22/25, 13/25 and 1/25 figures below are our sums of the paper's per-scene entries.

Table 1. Static grasping success with estimated grasp poses tab. 1

Table 2. Grasp accuracy after object movement tab. 2

For F3RM and LERF-TOGO, Time is the full rebuild cost, which the paper says excludes about 2 minutes of image capture §4.2. For SparseGrasp, Time is the optimisation budget tested tab. 2.

Table 3. 2D IoU of semantic distillation tab. 3

Table 4. Reconstruction and update time tab. 4

Paper vs code

No official code.

Searched, in order:

1. The paper and its arXiv abstract page link no code. The only URL in the LaTeX source is a commented-out placeholder §1 and is not used.
2. The arXiv abstract page lists no code or project link for this paper.
3. A GitHub project page for SparseGrasp (GitHub Pages, michaelyu781.github.io/sparsegrasp.github.io) was found by search. It links the paper and a video, and no code. It is not used as code.
4. GitHub repository searches for "SparseGrasp", "sparse gaussian grasp" and "semantic gaussian splatting grasping" returned no repositories. A GitHub user search for "junqiu yu" also returned nothing.

Pitfalls

• Mask model conflict. §3.1 and the fig. 2 caption name FastSAM for masks. §4 says the masks come from SAM ViT-H. Pick one and test the other; the paper does not say which was used for the reported numbers.
• Index clash in eq. 2. The paper reuses i for the pixel and for the Gaussian in the sum. This card uses k for the Gaussian. Eq. 1 implies a depth-ordered sum over Gaussians, but the paper does not restate the sort.
• Text query path is not described. The paper does not say how a text embedding is compared with the Gaussian features, or whether the query is projected into the 16-dimensional PCA space. Without that, the language-to-Gaussian step cannot be reproduced from the text alone.
• PCA scope and storage. The fit scope (per scene or global) is not stated. A new query must use the same projection, so the PCA basis must be stored with the scene.
• Pixel targets for L_sem. Eq. 5 compares rendered features with F^sem_pca, which is defined per mask. The paper does not say how mask features become per-pixel targets. Using each pixel's mask average is the natural reading but is an inference.
• Feature width of the Gaussians. The number of channels stored per Gaussian (f_i) is not stated. The 16-dimensional PCA result suggests 16, but the paper does not say so.
• Overlap rule. Masks sorted from smallest to largest, first mask wins §3.1. The sort key is mask area; ties are not discussed.
• Feature extraction time. The ~180 ms figure is not clear as per image or per scene, and the paper does not say whether it includes DUSt3R or training.
• Time comparison is not like-for-like. The F3RM and LERF-TOGO times exclude about 2 minutes of image capture §4.2. Whether the SparseGrasp time includes its capture and feature steps is not stated.
• Render-and-compare is under-specified. The frame used for the mean feature is not stated. Neither is the cosine threshold, the number of steps, the optimiser or learning rate, the rotation parameterisation, or how d_pred is computed. MOG2 parameters are not given; the paper calls the algorithm "simple and good enough" §3.3.
• Update budget changes the result. The 50 ms budget gives 1 of 25 successes; 100 ms gives 13 of 25; 200 ms gives 22 of 25 tab. 2. Quote the budget with any success rate.
• Novel-view IoU. The paper says its method is strong on novel views §4.1, but the 17-view F3RM baseline is higher on novel views (0.75 against 0.71) in tab. 3. The 2D IoU ground truth and dataset are not stated.
• Scale of the real-world tests. Seven objects with 10 attempts each, and five scenes with five trials each. Baseline configurations for F3RM and LERF-TOGO are not described.
• Add and remove objects. The paper says the scene needs a refresh §4.3. Speed claims exclude that refresh cost.
• Grasp model details. Only "retrain GraspNet" with centre, scale and rotation is given. Changes to the network, loss, candidate selection and training schedule are not stated. The dataset is the GraspNet dataset; check its terms before reuse.
• Supplementary file. The source tree contains supp.tex. It still has a template author block and reports different numbers (for example, 180 s reconstruction and LangSplat "5h+"). It is not cited here because the paper does not clearly publish it with the main text. Treat its numbers as unverified.
• Calibrated poses are an input. The paper feeds calibrated camera poses to DUSt3R §4. A deployment without calibration is outside what the paper tests.

Lineage

• Builds on: 3D Gaussian Splatting (kerbl20233d) §3; DUSt3R (wang2024dust3r) §3.1; MaskCLIP (zhou2022extract) §3.1; FastSAM (zhao2023fast) or SAM §3.1 §4; GraspNet (fang2020graspnet) §3.4. No arXiv ids are given in the source.
• Closest card in this wiki: Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields. The paper cites it as Feature-3DGS (zhou2023feature) and describes it as using a lightweight decoder §3.2.
• Compared against: F3RM (shen2023Distilled) and LERF-TOGO (lerftogo2023) §4.1 §4.2; LangSplat (qin2023langsplat) §3.2 tab. 4.
• Related sparse-view card, not cited by the paper: SparseGS: Sparse View Synthesis using 3D Gaussian Splatting.
• Superseded by: not known from content in this wiki.