Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments

In plain words

Teaching a robot in the real world is slow and risky, so many teams train it first in a computer simulation. Existing simulators each make a trade-off: some draw pictures well but handle grasping poorly, some run physics quickly but cannot simulate soft objects or cameras well, and most make it hard to build a new task from scratch.

Orbit is a toolkit that sits on top of a modern NVIDIA simulator. It gives you a common way to describe the robot, its sensors, the objects in the scene and the goal of the task, and it runs many copies of the scene in parallel on a graphics card. It also ships ready-made robots, sensors, controllers and example tasks, including tasks with cloth and other soft objects. You can build scenes with code or in the simulator's interface.

It matters because a faster, more flexible simulator makes it cheaper to try new learning methods, collect demonstrations and move a policy from simulation to a real robot.

TL;DR

• Orbit is an open-source robot-learning framework built on NVIDIA Isaac Sim, with a world/agent split that mirrors the real world and the robot's software stack §3.
• It bundles many robots, sensors, motion generators and benchmark tasks, and wraps several RL libraries [abstract] §4 §5.1.
• Author-reported: about 10x rigid-body and about 3x deformable-body throughput over the frameworks they compared with §5.5.
• It is not a learning algorithm; there are no equations to implement. The implementable parts are the abstractions, the actuator and sensor timing, and the example pipelines.
• Check the Pitfalls: the throughput comparisons are conditional, the sim-to-real evidence is mostly qualitative, and the repository has moved on since the paper.

Problem

Robot simulators tend to trade off four things: fast and accurate physics, photo-realistic sensors, handling of deformable objects, and easy task authoring §1. Vision-oriented simulators render well but simplify contact; physics-oriented robotics simulators are fast for rigid bodies but lack rendering, deformables or ROS support §1. CPU-based physics engines need CPU clusters to parallelise §2. Learning frameworks often define only the environment and not the rest of the agent's software, which causes code duplication and makes switching implementations hard §3.

Method

The paper describes a framework, not a learning method. Its design has two top-level parts §3:

• A world: the robots, sensors, objects and visualization markers that exist on one stage, with the same role as the real world.
• An agent: a computation graph of nodes that turns raw observations into actions for the robot. Perception nodes transform data (for example, depth image to point cloud); action nodes transform commands (for example, task-space to joint-space) §3.

A task adds the goal, reward computation and reset logic. It can also contain some agent nodes. Learning for one node is then a graph cut through the agent's computation graph, so the same world can host different tasks §3.

Components of the world:

1. Scene construction. Scenes can be built procedurally in scripts, from scanned meshes, in the Isaac Sim GUI, or a mix §3.
2. Robots. A robot loads a USD asset. Included scripts convert URDF, OBJ and STL files to USD and add colliders and friction materials §3 §3 fn.. One interface covers a whole class of robots, such as legged robots §3.
3. Actuators. The joint controller passes commands through an actuator model before it sets joint position, velocity or torque targets. The paper names DC-motor and series-elastic-actuator models and names delays and torque saturation as effects to model §3. Actuator groups let a complex robot use different transmission models for different sub-groups fig. 3.
4. Sensors. Physics sensors (range, force, contact) and rendering sensors (RGB, depth, normals) share one interface. The framework configures only the sensors a task needs, which the paper argues avoids overhead in parallel scenes. Each sensor has its own timer and returns its last value between updates §3.
5. Objects. Objects are passive. Textures and physics properties can be randomized. Objects with parts that change state, such as a hinge that is stiff until a seal breaks, can be modelled §3.
6. Markers. Primitive shapes (axes, spheres, meshes) can be drawn in the simulator GUI, for example to show goal states or frames §3.

Components of the agent and learning pipeline:

7. Nodes and timing. Every node has its own update rate. Data moves synchronously between nodes in Python, which the paper says avoids service-client overhead §3.
8. Motion generators. GPU versions of differential inverse kinematics, operational-space control and joint-level control. CPU versions of RMP-Flow for fixed-arm manipulators and OCS2 for whole-body control of mobile manipulators. Pre-trained locomotion policies that track base-velocity commands §4.
9. Teleoperation devices. Keyboard, Xbox-style gamepad and SpaceMouse. The same path collects demonstrations §4 §5.2.
10. Learning wrappers. rl-games, RSL-rl and stable-baselines3 for RL; demonstrations can be stored in the robomimic format §5.1 §5.2.
11. Benchmark tasks. The set has 26 tasks: 11 rigid-object, 13 deformable-object and 2 locomotion §1. Tasks can switch robots, objects, observations and action spaces through configuration §1 §4.

The agent loop, following the timing rules in §3:

# Agent-world loop with per-node timers  [§3]
# Sensors and nodes update at their own rates; values are held between updates

def step(world, agent):
    # Advance the physics simulation by one step  [§3]
    world.simulate()

    # Each sensor refreshes only when its own timer is due  [§3]
    readings = {}
    for sensor in world.sensors:
        if sensor.timer_due():
            sensor.refresh()
        readings[sensor.name] = sensor.last_value

    # Pass data synchronously through the agent's nodes  [§3]
    # The paper does not state how the node order is chosen
    values = readings
    for node in agent.nodes:
        if node.timer_due():
            values = node.run(values)

    # Apply the final joint targets through the actuator models  [§3]
    return values

Real-robot bridge for the Franka arm. The paper names quintic interpolation and does not give the coefficients or the boundary handling §5.4:

# Upsample 60 Hz joint commands to 1000 Hz for the robot  [§5.4]
# Quintic segments between command knots; boundary handling not stated

def upsample_commands(knots_60hz, times_1000hz):
    # Fit one quintic segment between each pair of consecutive knots  [§5.4]
    segments = fit_quintic_segments(knots_60hz)
    # Sample the segments at the 1000 Hz control times  [§5.4]
    return sample(segments, times_1000hz)

No equations are given in the paper. No algorithm boxes are given either.

Hyperparameters

Few numeric settings are stated. Most learning settings are not stated because the paper does not propose a learning method.

Results

Author-reported. Setup: GPU workstation (RTX 3090) for throughput; Isaac Sim 2022.1 and 2022.2 for the comparison frameworks §5.5, footnote. Baselines are robosuite v1.4, ManiSkill2 v0.4, Isaac Gym Preview 4 and DEDO v0.1 [§5.5, footnote]. Numeric throughput values appear only in the plots, not in the text.

Imitation learning on Franka-LiftCube. Author-reported; 100 trials per cell. G changes the goal, I changes the initial state, Both changes both tab. 2.

Source: tab. 2.

Paper vs code

Checked against the official repository at code_commit (see Pitfalls for the version gap). The paper's project page links two repositories: the NVIDIA-Omniverse Orbit repository (cloned here) and isaac-sim/IsaacLab. The clone's LICENSE and README use the Isaac Lab name LICENSE README.md.

• Actuator models. Paper: DC-motor and series-elastic actuator models §3. Code: a DC-motor configuration exists actuator_pd_cfg.py, and learned actuator networks, LSTM and MLP actuator_net.py actuator_net.py. Searching the source for "series elastic", "SeriesElastic" and "elastic actuator" found no match, so the SEA model is not present in this version.
• ANYmal base-mass randomization. Paper: base mass 22 ± 5 kg on ANYmal-D §5.4. Code: an additive base-mass randomization of (−5, 5) kg in the anymal_c_direct contrib environment anymal_c_env_cfg.py. The nominal 22 kg does not appear in that file, and that environment is for ANYmal-C, not ANYmal-D. This is not the same setup as the paper.
• Real-robot ZMQ bridge and quintic upsampling. Paper: ZMQ command transfer and 60 → 1000 Hz quintic interpolation §5.4. Code: no ZMQ transport was found in the source. The word "quintic" appears only in a comment in an unrelated demo script scripted.py. The paper's real-robot pipeline is not in this version.
• Not checked. PPO settings, the throughput benchmark scripts and the exact simulation frequency and decimation values. The paper gives no values for most of these, so there was nothing to compare.

Pitfalls

• The code has moved on. The clone is at a recent commit of a repository that is now called Isaac Lab; its license header covers 2022–2026. The paper describes a 2022–2023 snapshot. Do not assume that APIs or defaults in this commit match the paper. If you need the paper's version, find the release it was tied to; this card does not identify it.
• Isaac Sim is a separate, proprietary dependency. The framework is BSD-3-Clause, but its README says Isaac Sim and some assets fall under the Isaac Sim license, except for Kit-less Newton workflows README.md. The paper's footnote says Isaac Sim is free with an individual license §6 fn.. Check those terms before any commercial use.
• Throughput comparisons are conditional. The frameworks were adapted to the same action space, simulation frequency and control decimation §5.5. The competitors ran on different versions listed in a footnote §5.5, footnote. Against IsaacGymEnvs the speed is on par, because both use the same physics engine fig. 11. The "10x" and "3x" figures are against CPU-vectorized frameworks and DEDO, respectively.
• Memory limits in the comparison. CPU-vectorized frameworks reportedly crash at around 200–300 environments because of memory §5.5. This is the basis for the scaling claim, so it is hard to reproduce on small machines.
• Mesh resolution trades accuracy for speed. Finer cloth meshes are more accurate but slower, and at higher point counts the run runs out of memory §5.5 fig. 12.
• Rendering is the bottleneck. Physics reaches up to 125,000 FPS, but camera rendering reaches 270 FPS for ten cameras at 640 × 480 §7. Vision-heavy training will be limited by rendering, not physics.
• Sim-to-real evidence is mostly qualitative. The paper reports no success rates for the Franka or ANYmal transfer, only that the deployments ran §5.4. Fidelity of the whole simulator (rendering, sensors, physics) is listed as future work §7.
• Unstated details an implementer will need. PPO hyperparameters (said to be the same across libraries) §5.1. Simulation frequency and decimation values §5.5. Actuator model parameters and how the actuator network was trained §3 §5.4. Push magnitudes and timing §5.4. Sensor update rates §3. Cloth mesh resolutions and beam material values used in the comparison §5.5 fig. 10.
• Counts do not reconcile. The abstract says 16 robotic platforms [abstract]. Section 4 lists 4 mobile platforms, 7 arms and 6 end-effectors, with no total §4. The introduction counts 26 benchmark tasks (11 + 13 + 2) §1, while the abstract says "more than 20" [abstract].
• Figure and table numbers are derived. Numbers were assigned in LaTeX source order, because no compiled PDF was available. Check them against the arXiv PDF before citing them elsewhere.
• Version. The source is arXiv v2, with the update dated 2024-02-16 [raw metadata]. Text in v1 was not compared.

Lineage

• Builds on: Isaac Gym and IsaacGymEnvs for GPU physics and RL environments §1 app. B; Omniverse Isaac Sim for rendering and physics §1; RMP-Flow and OCS2 as motion planners §4; PhysX 5 for physics §2.
• Compared against: robosuite, ManiSkill2, IsaacGymEnvs (rigid-body throughput) and DEDO (deformable throughput) §5.5; the broader comparison in tab. 1 also includes MetaWorld, DoorGym, RLBench, iGibson, Habitat 2.0, SoftGym, ThreeDWorld, ManipulatorThor and SAPIEN.
• Superseded by: not stated in this paper. The repository now carries the Isaac Lab name LICENSE; no page in this wiki covers it.