Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

In plain words

Training a robot with trial and error usually means running a physics simulator on the processor, then sending the results to a graphics card that runs the learning model, and copying data back and forth at every step. That shuttling becomes the bottleneck, so training needs large clusters of processors.

Isaac Gym keeps the whole loop on the graphics card. The simulator, the calculation of what the robot sees and is rewarded for, and the learning code all work on the same data in place. Thousands of copies of a robot can then run side by side on one machine, and training that once needed a cluster can run on one GPU. The simulator is also set up so that the robot's joints, forces and body positions can be read and changed as ordinary Python arrays.

TL;DR

• Isaac Gym is a GPU physics simulator (NVIDIA PhysX) with a Python tensor API, so state, observations, rewards and actions stay on the GPU during training §2.1 §2.3.
• Training uses PPO, vectorised over thousands of environments and run through rl_games, on a single NVIDIA A100 in the reported experiments §4.
• Author-reported speed-ups are large, for example Ant reward 3000 in about 20 s and Humanoid reward 5000 in under 4 min §6.1.1 §6.1.2.
• Several results are compared with OpenAI's Shadow Hand setup on a CPU cluster, but the comparison uses different hardware, algorithms and seed counts §6.4.1.

Problem

Conventional RL for robots runs physics on CPUs, which need large clusters. The authors cite about 30,000 CPU cores for a Rubik's Cube policy and 6,144 CPU cores plus 8 GPUs for OpenAI's Shadow Hand work §1. An earlier GPU-simulation approach still copied physics state back to the CPU for observations and rewards §1 §2.1. Each CPU–GPU transfer adds overhead, which can be large for big simulations §2.1.

The paper targets learning with very many environment instances in parallel, where the GPU can exploit its fine-grained parallelism only if everything stays on the device §2.1.

Method

Pipeline §2.1 fig. 3. Stepping the physics, computing observations and rewards, and applying actions all run on the GPU without copying large amounts of data between devices. Two additions to PhysX make this possible: the GPU simulation can run without fetching results to the CPU after each step, and a direct GPU API exposes the state buffers and accepts control inputs §2.1. The paper contrasts the traditional CPU-based experience loop with this design fig. 3.

Scene layout §2.1. On a GPU, the authors pack all environments into one physics scene rather than splitting them into several scenes, because the GPU benefits from many objects in one scene. Environments are simulated in parallel without interacting with each other §1.

Tensor API §2.3 fig. 4. Physics data sits in flat global buffers, not in per-actor objects. Python code receives them as tensors through wrapping utilities that share the native buffer without copying §2.3.1. Views with the environment index as one dimension make per-environment observation and reward code vectorised §2.3. The same Python code runs on CPU or GPU by changing a device flag §2.3.1.

State tensors §2.3.2 tab. 1. Actor root state (position, quaternion, linear and angular velocity, 13 values per actor) and DOF state (position and velocity per degree of freedom) can be read and set. Rigid body state, DOF forces, contact forces and force sensors are read-only, and Jacobian and mass matrices are provided for articulated actors §2.3.2 tab. 1. Setting the root state and DOF state tensors resets poses and velocities, and an index buffer allows resetting a subset of environments §2.3.2.

Control tensors §2.3.3 tab. 2. Inputs are DOF actuation forces, PD position targets, PD velocity targets, and rigid-body forces and torques. PD targets apply only to DOFs configured with position or velocity drives, whose stiffness and damping are set through a separate API §2.3.3.

Physics §3. Robots are reduced-coordinate articulations in PhysX. The TGS solver is used, which runs sub-steps with one Gauss–Seidel iteration each and accumulates velocity deltas, giving convergence close to sub-stepping at lower cost per iteration §3. Tunable parameters are listed in tab. 3.

Training algorithm §4. All environments use PPO §4, run through rl_games, an end-to-end GPU implementation that vectorises observations and actions on the GPU §4. The paper gives no pseudocode for PPO itself. The loop below combines only the settings the paper states; rl_games internals are not described in the paper.

# PPO on GPU tensors, per iteration  [§2.1] [§4] [tab. 17]
# num_envs: 4096 to 16384 (per task, see tab. 17)

def train_iteration(env, policy, critic, optimizer):
    # Roll out horizon_length steps in all environments at once  [§4]
    observations = env.current_observations()    # (num_envs, obs_dim)
    for step in range(horizon_length):
        actions = policy.sample(observations)    # (num_envs, act_dim)
        # Physics, rewards and resets stay on the GPU  [§2.1] [§2.3]
        observations, rewards, dones = env.step(actions)
        store(observations, actions, rewards, dones)
    # Advantages with GAE, lambda = 0.95  [§4]
    advantages = gae(rewards, critic(stored_obs), dones, gamma=0.99, lam=0.95)
    # Clipped PPO update, several epochs over minibatches  [§4] [tab. 17]
    for epoch in range(mini_epochs):
        for minibatch in shuffle_into_minibatches(stored, minibatch_size):
            # Clip ratio epsilon = 0.2  [§4]
            update(policy, critic, optimizer, minibatch, clip=0.2)
    # Adaptive learning rate driven by a per-task KL target  [§4] [tab. 17]
    learning_rate = adapt_to_kl(learning_rate, measured_kl, kl_threshold)

Ant and Humanoid reward app. A.2.1. Both environments use one reward. It sums a progress term (the drop in distance to the target between steps), an upright bonus, a heading term, an action-size penalty and an effort penalty. A death term applies when the torso falls below the termination height app. A.2.1. The paper's formulas are unnumbered in the appendix. The implementation is in the code, see the reward function in the Paper vs code section.

# Ant / Humanoid reward, paper form  [app. A.2.1]
# potential = -distance_to_target / dt, computed each step

def ant_reward(state, actions, prev_potential):
    # Progress: change in potential since the last step  [app. A.2.1]
    progress = potential(state) - prev_potential
    # Upright bonus when torso up-vector . world-up > 0.93  [app. A.2.1]
    upright = 1.0 if torso_up_dot > 0.93 else 0.0
    # Heading: full weight at normalised angle >= 0.8, else linear  [app. A.2.1]
    heading = 1.0 if angle_norm >= 0.8 else angle_norm / 0.8
    # Action penalty, sum of squared actions; weight not stated  [app. A.2.1]
    action_penalty = sum(actions ** 2)
    # Effort, actions x motor strength x DOF velocity  [app. A.2.1]
    effort = sum(actions * motor_strength * dof_velocity)
    # Alive bonus only while torso is above termination height; weight not
    # stated  [app. A.2.1]
    alive = alive_weight * (torso_height >= termination_height)
    # Death term only while torso is at or below termination height; weight
    # not stated  [app. A.2.1]
    death = death_weight * (torso_height <= termination_height)
    # R_dof is listed in the paper's formula but not defined in its text
    return (progress + alive + upright + heading + effort
            - action_penalty + death)

Simulation setup per environment tab. 4. The table lists control type, simulation dt and control dt per task:

tab. 4 §4

Environments §4 §6. The paper uses Ant, Humanoid, Ingenuity, ANYmal (flat and rough terrain), Humanoid character animation with AMP, Franka cube stacking, and three in-hand manipulation tasks (Shadow Hand, Allegro Hand, TriFinger) §4 §6. Ant and Humanoid are described as MuJoCo-style benchmarks, while the hand tasks are presented as the main test of contact-rich physics §4.

Shadow Hand OpenAI variant §6.4.1. The policy receives only observations that could be measured on a real robot, while the value function also gets privileged simulator information (asymmetric actor–critic). Domain randomisation follows the OpenAI setup §6.4.1 app. A.4.1. The authors list differences from OpenAI: a tighter tolerance of 0.1 rad in addition to 0.4 rad, continuous rather than discrete actions, five seeds instead of one, no action delay or motor backlash in the randomisations, a different LSTM layout, and a different reward §6.4.1.

Hyperparameters

Author-reported PPO settings per training task tab. 17 app. A.3. Hidden units use ELU everywhere except AMP, which uses ReLU tab. 17. The adaptive learning rate uses a KL target per task, except AMP, which uses a fixed learning rate of 2e-5 and a KL threshold of 2e-1 tab. 17.

tab. 17 app. A.3. Allegro shares the Shadow Hand settings tab. 17.

General settings stated in the text §4:

• Discount γ = 0.99 for feed-forward networks, γ = 0.998 for LSTM networks §4.
• GAE λ = 0.95 and PPO clipping ε = 0.2 §4.
• Symmetric actor–critic (shared observations and network) by default; asymmetric for Shadow Hand OpenAI and TriFinger §4.
• Shadow Hand OpenAI LSTM uses sequence length 4 §6.4.1 tab. 17.
• The Shadow Hand OpenAI LSTM uses a 1024-unit LSTM followed by a 512-unit MLP §6.4.1 tab. 17.

Domain randomisation for Shadow Hand OpenAI app. A.4.1 tab. 18:

Other randomisation details app. A.4.1: random forces on the object with a per-episode probability drawn log-uniformly from 0.1% to 10%, a Gaussian force with standard deviation 1 m/s² times the object mass per axis, decayed by 0.99 every 50 ms; physics parameters are re-randomised on each reset with at least 720 steps between resets app. A.4.1.

Results

Author-reported. Setup: a single NVIDIA A100 and a 3.7 GHz Intel i7-8700K unless stated otherwise; training curves average 5 seeds, except where OpenAI's numbers are quoted from a single seed §4 §6.4.1.

Throughput §5: Ant peaks at about 700K environment steps/s, Humanoid at about 200K, and Shadow Hand at about 150K at 16384 agents. The paper states that increasing environments beyond the best setting did not help Ant or Humanoid §5 §6.1.2.

Paper vs code

Code: the official isaac-sim/IsaacGymEnvs repository at commit aeed298638a1f7b5421b38f5f3cc2d1079b6d9c3, BSD-3-Clause. The repository's README cites this paper as its technical paper and links the NVIDIA Isaac Gym page. The simulator itself is not in this repository; it is a separate NVIDIA download [abstract]. Ownership: the organisation isaac-sim belongs to NVIDIA, the authors' employer, and the repository copyright is NVIDIA's. This is the official example-environment code, but it is not a repository of the authors' own lab.

Searched for official code: links in the paper, abstract and project page (sites.google.com/view/isaacgym-nvidia, which links only NVIDIA's download page), then a GitHub repository search for isaacgymenvs, which listed the isaac-sim/IsaacGymEnvs repository alongside many forks.

Hyperparameter comparison (paper tab. 17 vs code at the commit above). Ant, Humanoid and Shadow Hand Standard agree on every value in tab. 17.

TriFinger is not in the paper's hyperparameter table, so these code values cannot be checked against the paper: learning rate 3e-4 constant, horizon 8, four epochs, minibatch equal to the number of actors, actor network 256-256-128-128 [isaacgymenvs/cfg/train/TrifingerPPO.yaml#L22, #L45-L46, #L59-L61], 16384 environments [isaacgymenvs/cfg/task/Trifinger.yaml#L11].

Other checks:

• Asymmetric LSTM ordering matches the text: the Shadow Hand OpenAI LSTM applies the recurrent layer before the MLP (before_mlp: True) [isaacgymenvs/cfg/train/ShadowHandPPOAsymmLSTM.yaml#L37], as the paper's §6.4.1 bullet describes.
• Ant reward: the code's weights are not in the paper: heading 0.5, upright 0.1, action cost 0.005, energy cost 0.05, death cost −2.0 [isaacgymenvs/cfg/task/Ant.yaml#L19-L28]. The upright and heading thresholds match the paper's 0.93 and 0.8 [isaacgymenvs/tasks/ant.py#L346, #L350].
• Ant alive term: the paper gates the alive bonus by torso height. The code always adds 0.5 [isaacgymenvs/tasks/ant.py#L358] and replaces the whole reward with the death cost when the torso is below the termination height [isaacgymenvs/tasks/ant.py#L365]. The two forms differ for fallen states.
• Ant effort term: the paper's effort term uses motor strength and DOF velocity. The code uses the absolute value of actions times observation entries 20–28 [isaacgymenvs/tasks/ant.py#L354]. Whether those entries are DOF velocity scaled by motor strength was not verified.
• Reward scale: most configs multiply the environment reward by 0.01 before PPO (for example [isaacgymenvs/cfg/train/AntPPO.yaml#L48-L49]). Franka and some others use 1.0 [isaacgymenvs/cfg/train/FrankaCubeStackPPO.yaml]. The paper does not say whether its reward thresholds (3000, 5000, 7000) refer to the scaled or raw reward.
• Simulation dt: reading dt divided by substeps as the physics step, Ant and Humanoid give 1/60 ÷ 2 = 1/120 s [isaacgymenvs/cfg/task/Ant.yaml#L43-L44], Ingenuity [isaacgymenvs/cfg/task/Ingenuity.yaml#L20-L21], TriFinger [isaacgymenvs/cfg/task/Trifinger.yaml#L61-L62] and Shadow Hand [isaacgymenvs/cfg/task/ShadowHand.yaml#L172-L173] match tab. 4. ANYmal uses dt 0.02 with 2 substeps, which gives 1/100 s, not the paper's 1/200 s [isaacgymenvs/cfg/task/Anymal.yaml#L82-L83]. Franka uses dt 1/60 with 2 substeps, which gives 1/120 s, not the paper's 1/60 s [isaacgymenvs/cfg/task/FrankaCubeStack.yaml#L40-L41]. This reading is an inference from the config fields.
• Not verified against code: the Shadow Hand, Allegro and TriFinger observation and reward definitions (appendix tables and reward listings), and the rl_games PPO implementation, which is an external library and not in this repository.

Pitfalls

• The simulator is not in this repository. The paper's simulator is distributed by NVIDIA as a download from the developer page. The BSD-3-Clause license covers IsaacGymEnvs only. The simulator's own license is not covered by this card and must be checked separately before use.
• The repository is archived. The GitHub page showed it as archived and read-only at the time of compile, so later fixes will not appear there.
• Table and text disagree for ANYmal and Franka. The table lists 8192 ANYmal environments but the text says 4096, and the Franka text says 16384 while the code uses 8192. The paper alone cannot settle which value is intended, so both are recorded here.
• Several PPO settings are unstated in the paper. The initial learning rate of the adaptive schedule, the rl_games adaptation rule, the entropy and gradient-norm settings, and the critic coefficient all appear only in the code. The code sets grad_norm 1.0 and entropy_coef 0.0 for Ant, for example [isaacgymenvs/cfg/train/AntPPO.yaml#L62], which the paper does not mention.
• Two keys for LSTM sequence length. ShadowHandPPOAsymmLSTM.yaml uses seq_length at [isaacgymenvs/cfg/train/ShadowHandPPOAsymmLSTM.yaml#L75] while other configs use seq_len. Which key rl_games reads was not checked.
• Central value blocks repeat values. central_value_config blocks in the Shadow Hand configs set their own minibatch and epoch values [isaacgymenvs/cfg/train/ShadowHandPPOAsymm.yaml#L73-L82]. An implementer should not confuse these with the policy's settings.
• Speed-up comparisons mix hardware and algorithms. The "2–3 orders of magnitude" and "300x" figures compare a single A100 with CPU clusters or CPU-based implementations, and the OpenAI numbers come from a single seed §6.4.1. The paper does not run iso-compute experiments on the same hardware.
• Horizon and environment count are paired. The best Ant setting is 8192 environments with horizon 16 fig. 5. For Humanoid, going beyond 4096 environments at horizon 32 gave no further gains and sub-optimal gaits, which the authors attribute to the short horizon §5 fig. 6. Using a table row's horizon with a different environment count is not what the paper tested.
• Results are author-reported and single-platform. Reward thresholds and wall-clock times depend on the GPU, the seed and the reward scale.
• Environment definitions are in the appendix. The observation and reward listings for the hand tasks and for ANYmal are in the appendix tables app. A.2 and were not all checked against the repository.

Lineage

• Builds on: GPU-based RL physics for robots from the authors' earlier CoRL 2018 work, which still copied state back to the CPU §1 §2.1; NVIDIA PhysX as the physics backend §3; the TGS solver §3; PPO §4; rl_games for GPU-vectorised PPO §4; AMP for humanoid imitation §6.2; OpenAI's Shadow Hand dexterity setup for the domain-randomised hand task §6.4.1. These works are named as this paper cites them; no content from them is used on this card.
• Compared against: OpenAI's CPU-cluster Shadow Hand training §6.4.1; the PyBullet AMP implementation of Peng et al. §6.2; MuJoCo CPU simulation in the introduction §1.
• Related concept pages: [robot-simulation], [policy-gradient], [domain-randomization], [sim-to-real-transfer].