Rearrangement: A Challenge for Embodied AI

In plain words

Most research on robots and embodied agents picks one narrow skill, such as walking to a point, and measures it. This paper proposes a shared task that covers many skills at once: rearrangement. The agent starts in some physical scene and has to change it until it matches a goal, like clearing a table or putting groceries on a shelf. The goal can be described by object positions, shown in a picture, given as a sentence, or taught by letting the agent look at the finished scene first.

The main idea is how to score such a task. Count the share of objects that end up where they should, and give nothing if the agent has also disturbed other things in the room. Efficiency numbers such as time, energy and computing cost are reported next to that score. To make this concrete, the paper describes five test setups in four simulators, from two robot arms sweeping cubes into a bin to a mobile agent tidying a house.

The paper does not present a new agent and reports no results. Its value is a common definition and a set of benchmarks that others can build on.

TL;DR

• A position and benchmark paper: it defines rearrangement as the task of bringing a physical environment into a specified goal state §3.
• Primary metric: task completion, the percentage of binary per-object tests passed, scored as zero if the agent harms other objects §4.1.
• Secondary metrics (simulated time, simulated energy, computation) are reported separately §4.2.
• Five testbeds in SAPIEN, RLBench, AI2-THOR and Habitat, with metric details in the appendix §5 app. A.
• No agent results are reported. Several linked code locations are missing, outdated or not authored by the paper's team (see Paper vs code).

Problem

Embodied AI has no shared standard task that spans simulators and lets different methods be compared, unlike image classification or language modeling in their fields §1. Existing work is either narrow, such as grasping or navigation benchmarks, or it compares methods that assume different inputs, such as known object poses in task-and-motion planners §2. The authors want end-to-end evaluation from raw sensor input to actuation, with no privileged information such as 3D models or ground-truth poses at test time §1 §3.

Method

The paper defines the task formally as a goal-conditioned partially observable process over rigid-body poses. The agent sees only observations and a goal specification. The goal is produced from the start state and a set of acceptable goal states §3. The rest of the paper fixes the parts an implementer needs to make a benchmark: goal types, embodiment and sensor levels, complexity axes, generalization settings, and the scoring rule §3 §4.

1. Describe the world as a set of rigid bodies. Each object or part has a pose in SE(3), and the world state is the product of these pose spaces. Joints are not enforced in this product, so not every combination is reachable §3.

2. Define the initial state s0 and the set S* of acceptable goal states. S* is the set of states in which all predicates P_i hold §3.

3. Build the goal g = φ(s0, S*) in one of five forms §3:

   • GeometricGoal: target poses or bounding boxes for the moved objects.
   • ImageGoal: a rendering of the goal state, which may be underspecified.
   • LanguageGoal: a text description of the goal state; it needs a checker for evaluation.
   • ExperienceGoal: the agent first sees the goal state and must restore it.
   • PredicateGoal: a set of predicates over objects, which other goal types can be converted into.

4. Choose the embodiment. The paper assumes a holonomic mobile base §3.2. Manipulation can be a magic pointer, a kinematic arm with abstracted grasping, or full physical simulation §3.1. The paper recommends the most realistic option that is practical §3.1.

5. Choose the sensors. Options are ground-truth poses (with realistic noise), intermediate representations such as segmentation and depth, or full simulated RGB, depth, LiDAR and other sensors. The paper recommends full simulated perception as the default §3.1.

6. Place the task on the generalization spectrum. The default is strong generalization: new objects and new environments at test time. Authors should state how much prior knowledge, such as CAD models or grasp demonstrations, overlaps with the test environment §3.3.

7. Score each episode with task completion §4.1. Use the pseudocode below.

8. Report secondary efficiency metrics separately, not as one combined number §4.2.

# Task completion for one episode  [§4.1]
# Input:  goal_poses, final_poses: one pose per task object
#         tolerances: one threshold per task object
#         harmed: True if any non-task object was moved
# Output: score in [0, 1]

def task_completion(goal_poses, final_poses, tolerances, harmed):
    # Do-no-harm gate: any harm gives a completion score of zero  [§4.1]
    if harmed:
        return 0.0
    passed = 0
    for goal, final, tol in zip(goal_poses, final_poses, tolerances):
        # Binary test per object, not an averaged error  [§4.1]
        # Test is task-specific, e.g. distance <= tol or IoU >= tol
        if passes(goal, final, tol):
            passed += 1
    return passed / len(goal_poses)

The paper's key formulas are below. The state space and the goal function are stated in the text §3 and are not numbered in the paper, so cite them by section.

\mathcal{S} = \mathcal{S}_1 \times \dots \times \mathcal{S}_n, \quad \mathcal{S}_i = SE(3) §3

g = \phi(s_0, S^*) §3

\mathrm{SPL} = S \cdot \frac{l}{\max(l_a, l)} app. A.5

The five testbeds are summarised in the paper's only table. Entries are copied as printed, including the spelling "Spate" tab. 1:

Testbed descriptions: T1 §5 app. A.1; T2 §5 app. A.2; T3 §5 app. A.3; T4 §5 app. A.4 fig. 12; T5 §5 app. A.5 fig. 13.

The AI2-THOR (T4) predicate, as the paper describes it, is a conjunction of two tests. The paper's wording does not define how the openness test applies to objects without an open/close state:

# AI2-THOR room rearrangement, one scenario  [app. A.4]
# Input:  for each changed object: agent_box, goal_box, openness, goal_openness
# Output: fraction of satisfied predicates in [0, 1]

def scenario_score(changed_objects):
    # One predicate per changed object  [app. A.4]
    passed = 0
    for obj in changed_objects:
        # (a) IoU of the 3D bounding boxes above 50%  [app. A.4]
        iou_ok = iou(obj.agent_box, obj.goal_box) > 0.5
        # (b) open/close state within 20% of the ground truth  [app. A.4]
        open_ok = abs(obj.openness - obj.goal_openness) <= 0.2
        if iou_ok and open_ok:
            passed += 1
    return passed / len(changed_objects)

The Habitat (T5) efficiency metric is episodic SPL app. A.5:

# Episodic SPL, Habitat  [app. A.5]
# success: 1 if the episode is successful, else 0
# shortest_len: shortest-path length l for the episode  [app. A.5]
# agent_len: length of the agent's path l_a (units not stated)

def episodic_spl(success, shortest_len, agent_len):
    # SPL = S * l / max(l_a, l)  [app. A.5]
    longest = max(agent_len, shortest_len)
    return success * shortest_len / longest

Hyperparameters

Values are as stated in the paper. Nothing here is a training hyperparameter, since the paper trains no model.

Results

Author-reported. The paper reports no performance numbers for any method. It defines the task, metrics and testbeds, and it argues that the field needs them §1 §5 §7.

Paper vs code

Checked code: allenai/ai2thor-rearrangement at commit 7fcf881 (Apache-2.0), linked from the paper's AI2-THOR project page. This is the 2023 AI2-THOR Rearrangement Challenge code, not the 2020 release that the paper describes README.md. The paper's version is dataset v0.1, while the repo ships later data folders (2021, 2022, 2023) and uses AI2-THOR 5.0.0 README.md.

Differences between the paper and this code:

• Predicate versus energy. The paper's AI2-THOR score is a binary predicate per object (IoU above 50% and openness within 20%) app. A.4. The thresholds match the code's constants rearrange_constants.py. The code, however, scores pose error as a continuous energy: half from IoU and half from corner distance, with a binary openness test environment.py. The binary form is in are_poses_equal environment.py.
• Phases. The paper describes one exploration phase of 1000 steps app. A.4. The code runs a walkthrough phase of 250 steps and an unshuffle phase of 500 steps rearrange_base.py.
• Do-no-harm. The paper makes an episode unsuccessful if the agent changes objects that were not changed in the start or goal state app. A.4. The code counts changed objects where the change energy is above zero tasks.py. The code does not show a gate that makes the score zero, so this was not confirmed.
• Grid step. The paper gives no grid size for the navigation grid app. A.4. The code uses 0.25 m constants.py.

Testbeds without a checked official repo:

• T5 (Habitat): the paper's link https://dexter1691.github.io/projects/rearrangement/ returned HTTP 404. A GitHub repository search for "habitat rearrangement" under the facebookresearch organisation returned no results. No code is recorded for T5.
• T1 (SAPIEN): the paper links https://github.com/haosulab/CSE291-G00-SP20 (MIT). It is a course project with main.py, final_env.py, baseline.py and solution.py. It was read through WebFetch only, with no line citations, and it is not recorded as the card's code because it is a course project and not the paper's testbed code.
• T3 (RLBench): the paper links https://github.com/stepjam/RLBench, and the linked project page points to examples/rearrangement_challenge.py there. The owner is the RLBench author, who is not an author of this paper, so it is not used.
• T2 (table organisation): ocrtoc.org returned no content through WebFetch. Not verified.

Discrepancies between the paper's text and its own tables:

• Scope. The introduction excludes within-object transformations such as melting and cutting §1. The discussion says object-state transformations, such as pouring or turning on an oven, are not addressed in this formulation §7. T4 nevertheless includes opening and closing, and a state-change action tab. 1 §5 app. A.4.

Pitfalls

• No method to implement. This paper defines a task, metrics and testbeds. Implementing it means building benchmarks, not reproducing results.
• Per-object tolerances are not stated. The paper says tolerances are task-specific and may be kept private so that the agent must infer them §4.1. Any value you need for a given task is "not stated".
• Do-no-harm differs by testbed. The general rule makes the score zero §4.1. The AI2-THOR section checks changed objects app. A.4. The Habitat metric has no harm check and uses a 1 m success radius app. A.5.
• "Within 20%" is ambiguous. The paper does not say whether the openness difference is absolute or relative, nor how the 20% is measured app. A.4. The code compares an openness difference with the same tolerance environment.py; the difference's definition is not shown here.
• The AI2-THOR conjunction is undefined for some objects. The paper requires both IoU and openness app. A.4. For objects without an open/close state, the openness test has no meaning and the paper does not say what replaces it. The code uses one test or the other per object environment.py.
• Typos and conflicts in the table. Table 1 writes "Spate" for the T1 manipulator tab. 1. The appendix describes spade or spatula end effectors with joint-level control app. A.1. Table 1 lists Grab/Release for T1. The paper does not reconcile these.
• Units for SPL are not stated. The paper does not say whether path lengths are in metres or steps app. A.5. Shortest paths come from a generalised travelling-salesman formulation solved with OR-Tools; the goal ordering and the formulation details are not stated app. A.5.
• Efficiency metrics are suggested, not defined. Simulated energy is described as integrated virtual work, with no formula. FLOPs and real-time factor are suggested but not specified §4.2.
• Exploration length is a design choice. The AI2-THOR setup allows 1000 steps app. A.4; the code's 2023 phases use 250 and 500 steps rearrange_base.py. Results from one setup do not carry over to the other.
• Simulator mix. Testbeds use four simulators, two of them externally maintained. Code for T2, T3 and T5 is not in one place and is partly unverified (see Paper vs code).
• Sim-to-real is left open. The paper recommends real-robot variants and says automatic evaluation on real robots is hard §4.1 §7.

Lineage

• Builds on: none as a method. It contrasts itself with task-and-motion planning §2 and with navigation benchmarks such as PointGoal, which it says the mobile variant generalises §7.
• Compared against: navigation benchmarks §2 §7; rearrangement planners that assume known object poses §2.
• Superseded by: not stated. The AI2-THOR code linked here is the 2023 challenge version README.md, so it differs from the 2020 testbed.