iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes

In plain words

Teaching a robot to tidy a house by trial and error is slow and risky in the real world. iGibson 1.0 is a virtual house for that job. It offers 15 detailed, furnished home layouts whose furniture can be opened, pushed and moved, so a simulated robot can practise real physical interactions rather than only walking around.

The virtual robot sees the world through simulated cameras and laser range sensors. The simulator also changes the look and physical feel of objects, and swaps in different objects of the same type, so a learned skill has to cope with variety instead of memorising one scene. There are built-in path planners for moving arms and bases without collisions, and a mouse-driven interface lets a person demonstrate a task, which can then be copied by a learning algorithm.

The authors test three things: whether this variety makes navigation skills more robust, whether human demonstrations are enough to teach manipulation, and whether pretraining on which parts of a scene can be pushed speeds up later learning. A navigation policy trained only in simulation was tried on a real robot, to see whether the simulated sensors are realistic enough.

TL;DR

iGibson 1.0 is an open-source simulator for robot tasks that involve touching and moving objects, in large home-sized scenes. It provides physics, rendering of several sensor types, domain randomization, motion planning and a demonstration interface. The paper's experiments are author-reported and small: randomization improves generalization of navigation policies in held-out scenes, and imitation learning works on demonstrations collected through the interface. The official repository has moved on to version 2.2.2, so the 1.0 settings must be read from this card, not from the current code.

Problem

Simulators that model physics and robots usually cover narrow tasks and small, clean scenes. Simulators with large homes either allow navigation only, or offer simplified interaction (for example, object swaps or teleport-style actions) §1. The authors argue that these cannot support end-to-end sensorimotor loops for tasks that need continuous interaction with the scene, and that simplified interaction makes transfer to real robots hard §1. The paper's comparison table lists iGibson 1.0 against Gibson, Habitat, SAPIEN, AI2-THOR, VirtualHome and TDW tab. I §2.

Method

The paper describes a software system rather than a learning method. The components are listed below, with the section that defines each.

1. Environment API. Follows the OpenAI Gym convention: an action goes in, and an observation, reward and end-of-episode flag come out. Behaviour is set by config files. A config creates an Environment holding a Task (reward, initial and final conditions) and a Simulator. The simulator holds a Scene, interactive Objects, one or more Robots and a Renderer §3.1.
2. Scenes. Fifteen scenes are built from the 3D reconstructions of the Gibson v1 dataset §3.2. Scenes were annotated in two passes. First, layout: floors, walls, doors and window openings. Second, all objects: 3D bounding boxes with class labels, for 57 object classes §3.2. The authors state that they built each interactive scene from scratch rather than editing the original mesh, which avoids reconstruction artifacts §3.2. Object models come from ShapeNet, PartNet-Mobility and SketchFab, with physical materials and per-link collision meshes, masses from Amazon search medians of the top 20 results §3.2.
3. Extra scene sources. Floor plans from CubiCasa5K and layouts from 3D-Front are converted into further interactive scenes. The authors report 6,297 scenes from CubiCasa5K and 6,049 from 3D-Front after filtering app. Integration of CubiCasa5K app. Integration of 3D-Front.
4. Sensors. A physics-based renderer uses metallic, roughness, albedo and tangent-space normal maps, with light probes and shadow mapping §3.3 app. Physics Based Rendering. Virtual channels include RGB, depth, normals, point clouds, segmentation (semantic, instance, material, movable parts), optical and scene flow §3.3. LiDAR is simulated with 1 beam or 16 beams, plus a drop-out noise model and an optional conversion of 1D scans into bird's-eye occupancy maps §3.3.
5. Domain randomization. Object randomization replaces each labelled bounding box with another model of the same class, keeping class positions §3.4. Material randomization assigns a different material to each object part, and dynamics properties can follow the material through a curated mapping §3.4.
6. Motion planning. Sampling-based planners: RRT, bidirectional RRT and lazy PRM, followed by acceleration-bounded shortcutting §3.5.
7. Human interface. Mouse and keyboard control in a viewer window to navigate and push, pick and place objects. The interface is wired to the motion planner so that a user can command base or arm configurations §3.6. Demonstrations are then used for imitation learning §4.2.
8. Navigation experiments §4.1. PointGoal policies (depth, velocities, goal and the next 10 waypoints 0.2 m apart) and object-navigation policies (RGB, goal: target lamp fills at least 5% of the image) are trained with SAC in eleven scenes and tested in four held-out scenes with held-out textures. A LiDAR PointGoal policy (512-ray 1D scan, velocities and goal) is trained in one scene, Rs_int, and run on a real robot §4.1 app. Domain Randomization for Visual Navigation app. LiDAR-Based Point-to-Point Navigation.
9. Imitation learning §4.2. A behaviour-cloning policy for pick-and-place is trained on 50 human demonstrations; a mobile-manipulation policy is trained on 200 teleoperated demonstrations, using a system like RoboTurk §4.2.
10. Interaction pretraining §4.3. A U-Net is trained to predict, per pixel, whether a push at that point moves the object. Its output gates the Q-value map of a DQN policy for two drawer and cabinet closing tasks §4.3 app. Pretraining in Fully Interactive Scenes.

# Phase 1: label pixels by simulated pushes  [§4.3]
# [app. Pretraining in Fully Interactive Scenes]
def collect_push_labels(scene, n_locations=4000, n_pushes=10):
    labels = []
    # Sample pixels at random: n_locations per scene  [§4.3]
    for point in sample_pixels(scene, n_locations):
        for _ in range(n_pushes):
            # Push 30 cm opposite the surface normal, force cap 60 N  [§4.3]
            moved = push_point(scene, point, distance_m=0.30, max_force_n=60.0)
            # Success if the point moves more than 10 cm  [§4.3]
            labels.append((point, moved > 0.10))
    return labels

# Phase 2: gate DQN Q-values with the pushability mask  [§4.3] [app.]
def gated_q_values(rgb, unet, q_net):
    # rgb: (B, 3, 128, 128); mask: (B, 1, 128, 128)  [app.]
    interactable = unet(rgb)
    # Dense Q-values, same shape as the input image  [app.]
    q_map = q_net(rgb)
    # Keep Q only at pixels predicted interactable; zero the rest  [app.]
    gate = is_interactable(interactable)  # threshold not stated
    return q_map * gate

# Act: the pixel with the highest gated Q is pushed by a motion planner  [app.]
pixel = argmax(gated_q_values(rgb, unet, q_net))

Hyperparameters

Values are as stated in the paper. Where the paper gives no value, the row says "not stated".

Results

Author-reported. Setup: simulated scenes and a real apartment (Rs_int counterpart) for the LiDAR test, simulated Fetch robot for imitation learning, Locobot for the real robot. Hardware for training is not stated beyond "multi-GPU". Baselines are the same agents without the named feature.

The comparison with other simulators is a feature checklist, not a benchmark tab. I.

Paper vs code

Official code: StanfordVL/iGibson (MIT), linked from the authors' project page §1. The checked commit is 7d9598278b95ec1039e1f4366c66b45346521e44, which reports version 2.2.2 [code: raw/code/2012.02924/igibson/init.py#L7]. That is a later release than the paper's 1.0. Items were checked in this commit only.

• Physics and render rates match. Physics timestep 1/120 s and render timestep 1/30 s are the defaults simulator.py, and the loop runs physics_timestep_num physics steps per render simulator.py, as the paper describes app. Physics Simulator and Full Simulator Performance. No discrepancy found.
• LiDAR beam count differs. The paper uses a 512-ray scan §4.1 app. LiDAR-Based Point-to-Point Navigation. The static-navigation config for a Hokuyo sensor uses 228 horizontal rays, sub-sampled from 683 turtlebot_static_nav.yaml. This is a different config, and the LiDAR experiment's own config is not in the repo.
• Waypoints. The paper's PointGoal observation has 10 waypoints §4.1. The repo's fixed PointGoal task creates 10 visualisation markers point_nav_fixed_task.py. Whether the same 10 waypoints feed the policy was not verified.
• Renderer BRDF. The paper names Cook–Torrance for the specular pre-integration app. Shading Models. The shader's geometry term is Schlick–GGX with Smith's method spbrdf_cs.glsl. Both are common choices for that term; the paper's exact formulation was not checked against 1.0.
• Experiment training code is not in the repo. A search of the repo for the pretraining network, push task, PushDrawer or PushCabinet, found no match. SAC, DQN and behaviour-cloning training for the three experiments were not found. Their hyperparameters are therefore paper-only.

Pitfalls

• Version mismatch. The paper describes 1.0; the repo is 2.2.2. Config names, sensor APIs and scene data format may differ. Build from the paper values in this card, and read the repo only for behaviour the paper leaves out.
• Scene assets have their own licence. The project website says the datasets are encrypted and may only be used with iGibson [project page; not verified in the paper]. That is a restriction outside this paper's CC BY 4.0 grant. This card reuses no scene or object data.
• Number conflict. The text gives 31.25% success for the depth baseline §4.1. Table II gives 31.15% for the same row tab. II. The text and table both give 0.27 and 0.40 SPL.
• Appendix numbering. The appendix is an unnumbered \section*, so its subsections have no stable number. They are cited here by subsection title, e.g. app. Physics Based Rendering.
• Equation numbering. The Fresnel formula is an unnumbered display. It is cited as app. Shading Models, not by number.
• Mask versus heatmap. The main text calls the pretraining output a heatmap that "gates" Q-values §4.3. The appendix calls it a binary mask and zeroes Q at non-interactable pixels app. Pretraining in Fully Interactive Scenes. The threshold that makes a mask from the heatmap is not stated.
• Undefined push measurement. "A motion of the point for more than 10 cm" is the success test §4.3, but how the point's motion is measured (tracked pixel, 3-D point, or object) is not stated.
• Behaviour-cloning learning rate. The paper states Adam with learning rate 0.1 app. Imitation Learning: Human Demonstrated Manipulation. That is far above the 1e-4 and 5e-4 SAC rates in the same paper. The code is not available for this step, so the value could not be checked. Do not assume it is correct without a test run.
• Missing RL details. Action space and controller rate for PointGoal, reward functions, episode lengths, and SAC batch size and temperature are not stated. The LiDAR controller runs at 10 Hz app.; no other controller rate is given for navigation.
• Small evaluation sets. Manipulation IL is tested on 100 episodes with 5 unseen mugs. Mobile-manipulation IL is tested on 20 episodes §4.2. Expect wide error bars.
• Real-world test is one apartment. The LiDAR policy was tested on one real apartment with a Locobot and a single-beam Hokuyo app. LiDAR-Based Point-to-Point Navigation. The 9-percentage-point drop (33% to 24%) is for one scene pair, and failures were said to fall on the same episodes §4.1.
• Pretraining gains are qualitative. The text reports faster training with pretraining for two tasks, with curves only §4.3 fig. 7. No success rate or step count is given.
• Shadows do not match real light. The shadow map uses a uniform orthographic light along +z, which the authors say does not match real indoor light app. Shadow Mapping.
• Scene density. The authors report 75 objects per room in the 15 hand-built scenes, against 37 (3D-Front) and 32 (CubiCasa5K) app. Comparing iGibson, CubiCasa5K and 3D-Front Scenes. Expect fewer objects in the converted scenes.
• 3D-Front cleanup. Objects overlapping more than 80% of their volume were removed; smaller overlaps were fixed by shrinking objects by at most 20% app. Integration of 3D-Front. Other preprocessing choices are not stated.
• Cost. A simulator step is 4 physics steps plus one render pass app.. Mean full-step rate is 100 Hz with a robot, against 136 Hz without tab. V. High-fidelity rendering (512×512 with anti-aliasing and shadows) runs at 188 fps mean tab. IV.

Lineage

• Builds on: Gibson v1 3D reconstructions §3.2; RoboTurk-style teleoperation §4.2; sampling-based planners adapted from a PyBullet planner library §3.5; SAC app.; DQN §4.3; U-Net §4.3.
• Compared against: Gibson, Habitat, SAPIEN, AI2-THOR, VirtualHome and TDW in tab. I; the earlier Interactive Gibson benchmark, which segmented original meshes instead of building scenes from scratch §3.2.
• Superseded by: not stated in this wiki. The repository has moved to version 2.2.2, but this card does not cover that version.