Grasping Object: Challenges and Innovations in Robotics and Virtual Reality

In plain words

When you pick up a cup in a video game or a VR headset, the hand in the scene often passes through the cup, feels nothing, or sticks to it in an unnatural way. Getting this right is hard, because the computer has to model the hand, the object (soft objects are the hardest), what the user feels, and how the scene looks, all in real time.

This paper is a review. It explains what makes virtual grasping difficult, surveys the main ways researchers have tried to solve it, and compares those ways with how robots grasp real objects. The authors argue that the two fields have a lot to share, and they point to a few directions they think are worth pursuing. They do not build or test a system of their own.

TL;DR

• A survey of grasping in VR, set against robotic grasping §1.
• Main open problems: modelling soft fingers and deformable objects, penetration between hand and object, limited haptic hardware, and the lack of a common metric for VR grasping §2.1.
• Methods are grouped into physics-based, other visual, and data-driven approaches §2.2. The authors suggest that data-driven methods, combined with physical constraints and caging ideas, are a promising direction §4.
• No experiments, no new algorithm, and no code. Use it as background and a reading list, not as a source of implementable numbers.

Problem

The paper asks why stable, realistic grasping in VR is still an open problem. The authors name these obstacles §1, §2.1:

• Grasping depends on object properties such as mass, size, material and stiffness §1.
• Most existing techniques simplify the target to a rigid body, which fails for deformable objects §1, §2.1.
• Modelling the hand is hard. The hand has many joints, the thumb needs its own model, and the contact between skin and object is hard to reproduce §2.1.
• Virtual hands are not stopped by physical contact the way real hands are. This causes visible penetration between hand and object §2.1.
• Haptic hardware is limited: it is expensive, not portable, or cannot render detail. Controllers usually give only simple force feedback §2.1.
• Gravity is seldom simulated in haptic feedback §2.2.
• The field lacks a common metric or benchmark for VR grasp quality, especially for deformable objects §2.1. In robotics, the grasp wrench space is the reference metric for grasp quality §2.1, and the paper notes that NIST has published robot performance benchmarks §2.1.

Method

The paper is a narrative review. It has no algorithm, equations of its own, or experiments. Its contribution is the organisation of the material:

1. Challenges (VR). Three groups: high-fidelity modelling of the hand and contact §2.1; device performance (display, weight, battery, haptics, visual artefacts such as the screen door effect and lens distortion) §2.1 tab. 1 fig. 4 fig. 5 fig. 6; and the missing theoretical framework and metrics §2.1.
2. Grasping methods (VR). Three groups §2.2:
   • Physics-based: grasps are produced by simulating forces and contacts, with elastic-surface models and collision detection. Examples discussed are beam-based deformation models, haptic devices with vibration or brake feedback, and a particle-based contact method §2.2.1 fig. 7 fig. 8.
   • Other visual: snapping the object to the hand, freezing the hand on contact, and example-based animation. These look stable but feel unnatural §2.2.2 fig. 9. Deformation transfer and cage-based deformation control are also covered here. The authors note that cage-based control costs less to compute than force control §2.2.2.
   • Data-driven: grasp databases built from human demonstrations or simulation, plus machine learning or matching to choose a grasp. The authors say this needs a lot of data and efficient matching §2.2.3 fig. 11.
3. Robotic grasping. A short overview to compare with VR. Analytic methods use contact geometry and the force-closure and caging criteria §3.1. Data-driven methods learn from human operators or learn an association between object shape and hand shape, and some handle deformable or fragile objects §3.2 fig. 10 fig. 12.
4. Proposed directions (discussion). The authors group methods into physics-constraint methods, data-driven methods, and "robotic-like" grasping that combines the two §4 fig. 13. They suggest that caging grasping is a candidate for VR because it is stable without full immobilisation and does not need precise object models §4. They also suggest pairing caging-based grasping with cage-based deformation control for deformable bodies §4. Computer vision, particularly hand tracking and hybrid tracking, is discussed as an enabler §4.1.
5. Conclusion. Cross-disciplinary work (materials, robotics, kinematics, computer vision, human behaviour) is needed §5.

The paper's taxonomy of VR grasping methods, written as a plain data structure. This is a restatement of the categories in §2.2 and §4; the paper gives no algorithm for choosing among them.

# Taxonomy of VR grasping methods  [§2.2] [§4] [fig. 13]
# Each family lists the examples the paper discusses under it.

def grasping_taxonomy():
    # Physically simulated contact and forces  [§2.2.1]
    physics_based = [
        "elastic-surface deformation models",   # [§2.2.1] [fig. 7]
        "vibration or brake haptic feedback",   # [§2.2.1]
        "particle-based contact",               # [§2.2.1] [fig. 8]
    ]
    # Visual rules without full physics  [§2.2.2]
    other_visual = [
        "snap object to hand",                  # [§2.2.2] [fig. 9]
        "freeze hand on contact",               # [§2.2.2]
        "example-based animation",              # [§2.2.2]
        "cage-based deformation control",       # [§2.2.2] [§4]
    ]
    # Learned from human or simulated grasp data  [§2.2.3]
    data_driven = [
        "grasp database lookup",                # [§2.2.3] [fig. 11]
        "learned grasp space",                  # [§2.2.3]
    ]
    # Suggested direction: caging plus data-driven  [§4]
    suggested = ["caging-based grasping", "data-driven grasping"]
    return physics_based, other_visual, data_driven, suggested

Hyperparameters

Not applicable: the paper specifies no model, training procedure or experiment. The only numbers in the text are anatomical and device facts:

Results

Not applicable: no experiments. The paper's only quantitative table is a device specification table. Values as printed in the paper, and the paper attributes them to the web sources it cites (Wikipedia pages for each device) tab. 1:

The paper's point from this table is that battery life and portability remain problems tab. 1 §2.1. The table reflects the device models the paper lists, as of its 2024 version, and the figures may have changed since.

Paper vs code

No official code. The paper links no repository, project page or code availability statement, and the arXiv metadata has no comment field. A GitHub repository search for "grasping virtual reality survey" and a user search for the authors' names both returned nothing.

Internal inconsistencies the paper itself contains are listed under Pitfalls.

Pitfalls

• No method and no experiments. Nothing in this paper can be implemented or reproduced. Treat every statement as a summary of other people's work, and go to the cited source before building on it.
• Table 2 does not exist. The Discussion says "From Table 2" §4, but the source has only Table 1 tab. 1. The claim that data-driven methods will dominate is therefore not backed by any table in this paper.
• Figure 12 is cited for the wrong content. The Discussion cites Figure 12 for animation-based and geometric grasping §4. Figure 12 shows irreversible damage to a deformable object, from the robotic-grasping section §3.2 fig. 12. Figure 13 is the new classification fig. 13.
• Figure 11 appears before Figure 10 in the source. Cite them by the printed labels, not by their order in the file.
• Citation attribution is inconsistent. Some names in the text do not match the citation keys, for example the same reference (Gabardi2018) is credited to a "Mulatto et al." and to "Burton et al." §2.1. The caption of Figure 11 misspells "Goldfeder". "Opera et al." §2.2.2 is probably Oprea et al. (2019). Check the original paper before quoting an attribution.
• Dates and versions. The VR device statements ("just in the past year", Vision Pro) reflect 2024 and are out of date.
• Device table sources. tab. 1 relies on Wikipedia pages and the figures in it have no direct manufacturer citation. Verify before using the numbers.
• Gravity and haptics. The paper says most researchers model friction but not gravity in haptic rendering §2.2. This is a claim from the authors' reading of the field and is not quantified.
• Penetration and cost trade-offs. The paper says detecting collisions every frame is too costly for some applications and that games favour frame rate over exact collision §2.1. No numbers are given for either side.
• Diagram and image credits. The figures in the source are reproduced from other papers, and their captions credit those papers. Do not reuse any of them on this site; the card describes them in words only.

Lineage

• Builds on: none as a method. It surveys robotic grasping work §3, and recommends a separate robotic grasping survey for detail §3.
• Compared against: robotic grasping methods (analytic and data-driven) §3 and VR grasping methods §2.2.
• Superseded by: not known from content in this wiki.