Challenges and Outlook in Robotic Manipulation of Deformable Objects

In plain words

Most robot grippers and planners assume that the objects they handle keep their shape. Cloth, cables, fruit, bags and soft tissue bend, stretch and change shape as soon as they are touched, so those assumptions break down. This paper is a review of the field. It goes through the parts of a robot system that have to cope with this: the gripper and robot design, the sensors, the models of how the object deforms, the planning of moves, and the control of the motion. For each part it describes what has been done so far and what is still open.

The authors also give their own views. One is that a robot should sometimes plan to make contact with a soft object, because contact limits how the object can move, instead of always avoiding contact. Another is that sensing looks like the most promising area to push forward, based on a small poll of researchers. The paper proposes no new method, so there is no algorithm to implement from it directly.

TL;DR

• A survey and position paper on robotic manipulation of deformable objects (DOM), not a method §1.
• It splits a DOM system into five parts: gripper and robot design, sensing, modelling, planning and control §1 fig. 2.
• Each part gets a "current capability" review and a "challenges and outlook" discussion §2–§6.
• Authors' positions: plan to make contact rather than only avoid it §5.2; no single model is best, and physical models matter §8.
• The only data is a poll of 31 respondents rating importance and research maturity; sensing was rated the most promising area §1 fig. 3.

Problem

Robotic manipulation usually assumes rigid objects. The authors argue that rigidity is a valid assumption only when deformation can be neglected for the task, and that many objects robots must handle deform too much for that §1. They name three technical difficulties: perceiving how the object deforms, the very large number of degrees of freedom a soft body has, and the nonlinear behaviour of its deformation model §1. Application areas they list are dressing assistance, cable harnessing, harvesting fruit and vegetables, and surgery §1 fig. 1. The stated aim is to identify scientific challenges and future directions rather than to report progress, and previous work and open problems are given equal weight §1. The paper does not state any assumption about a specific object or setting beyond this scope.

Method

This is a survey. It proposes no algorithm, no equations and no model. The paper frames a DOM system as five parts that feed one another §1 fig. 2. The sketch below only follows that framing; it is not an implementation:

# Five-part framing of a DOM system  [§1] [fig. 2]
# Input:  an object that deforms when touched
# Output: a motion that achieves the task

def manipulate(object, task):
    # Gripper and robot design: hardware that holds the object  [§2]
    gripper = choose_gripper(object, task)

    # Sensing: estimate the current shape and contact  [§3]
    state = sense(gripper, object)

    # Modelling: relate sensor readings to robot motion  [§4]
    model = model_deformation(state)

    # Planning: pick a sequence of target configurations  [§5]
    plan = plan_configurations(model, task)

    # Control: turn the plan into robot inputs  [§6]
    return control(plan, model, state)

Its structure, with the authors' own section numbers:

1. Gripper and robot design §2. Current capability: deformable objects are usually handled with custom grippers, and the robot is mostly rigid §2.1 fig. 4. Outlook: compliant, human-like designs look promising; surgery needs task-specific grippers because of biocompatibility and limited space; soft, non-anthropomorphic grippers are little studied for DOM; a standard gripper plus tools would need reasoning about which tool to use §2.2 fig. 5.
2. Sensing §3. Current capability: vision gives the global shape, while force and tactile sensing give local shape and contact; vision-based tactile sensors can estimate stiffness; tactile sensing can also recover some global properties §3.1. Outlook: fuse vision, force and tactile data; choose a representation that is robust to noise and occlusion; use active perception to see occluded parts; trade off representation dimension against accuracy; end-to-end learning and demonstrations §3.2.
3. Modelling §4. Current capability: local models are Jacobians (called interaction matrices in visual servoing), updated online; global models are finite element models or neural networks §4.1. Outlook: simulators for DOM are limited (SoftGym, SOFA and ChainQueen are named), and a unified simulator for soft robots and soft objects may be needed §4.2. A hierarchy with a simple local model below and a learned global model above is proposed §4.2.
4. Planning §5. Current capability: configuration-space planning, energy-based planning, distributed receding-horizon planning for several robots, learning from demonstration and reinforcement learning §5.1. Outlook: use environmental contact to remove degrees of freedom; plan grasp points and regrasps; reason about semantic goals such as "folded" §5.2.
5. Control §6. Current capability: data-driven model predictive control for cutting, fuzzy compensation with impedance control in surgery, shape-from-template, and shape control for wire and cable assembly §6.1. Outlook: feedback control despite inaccurate models; offline model learning combined with online adaptation; handling under-actuation when several features are controlled; regaining contact automatically when it is lost §6.2.
6. Practical applications §7 fig. 6. Laundry folding, assistive dressing, surgical robotics, food production and retail, and marine robotics, each with its practical obstacles.
7. Summary §8 fig. 7. Key messages per area, summarised in one overview figure.

No table or equation in the compiled text is needed to follow the paper. A simulator comparison table exists in the source only as commented-out code and is not part of the published text (see Pitfalls).

Hyperparameters

Not applicable: the paper runs no training or experiment and gives no algorithm settings. The only numbers in the text describe the survey it reports:

Results

Author-reported. Setup: an online poll of people working in related fields, at various career stages, who rated the importance and research maturity of each of the five components on a scale of 1 to 4 §1. There are no datasets, models, baselines or hardware.

Paper vs code

No official code. The paper's only link is to the authors' survey form, not to code §1. Its arXiv metadata has no comment or code link. No project page is linked. A GitHub repository search for "deformable object manipulation survey robotics" returned no repositories.

Pitfalls

• It is not a method. An implementer cannot take an algorithm from this paper. Anything to build has to come from the works it cites, and those have no cards here.
• Authors' views are not findings. Statements such as "we argue" §5.2, "we believe" §1 §8 and "all models are wrong, some are useful" §8 express the authors' position. They are not supported by experiments in the paper.
• The sensing ranking rests on a small poll plus reasoning. The text gives the reasons (the deep-learning trend and sensing as a prerequisite for the other steps) §1. The poll's means and variances appear only in the chart fig. 3; the text does not state the numbers, the sampling method, or how many respondents were in each group.
• Version. The source is arXiv v2, dated 2021-12-14. Its header says the paper was conditionally accepted for IEEE RA-M, but the source does not say whether the journal version differs. Check the published version before quoting exact wording or figure numbers.
• Stale scope. The literature review ends around the 2021 revision. The paper does not say anything about later work, and this card does not add any.
• Figure numbers are inferred. The figure numbers above follow the order of figure environments in the LaTeX source. The compiled PDF was not checked. No figure is reproduced on this page.
• Commented-out text is not used. The source contains commented-out material (for example a simulator comparison table and a 3C manufacturing paragraph) that is not part of the compiled paper. This card does not use it.

Lineage

• Builds on earlier surveys. The authors place their work against Jimenez (model-based planning), Herguedas et al. (multi-robot systems), Nadon et al. (multi-modal sensing), Arriola et al. (modelling), Sanchez et al. (classification by physical property) and Yin et al. (modelling, learning, perception and control) §1. None of these has a card in this wiki.
• Compared against: the authors state that their focus is on scientific challenges and future directions rather than on progress reports, and that they also discuss practical application challenges §1.
• Superseded by: not stated.