Robot simulation for learning

Overview

Teaching a robot on hardware is slow and risky, so learning pipelines often train the policy in simulation first. A simulator has to balance fast and accurate physics, realistic sensors, deformable objects and easy task authoring. The designs on offer trade these off against each other 2301.04195 §1.

The design the paper proposes splits the simulation in two. A world holds the robots, sensors, objects and markers, and plays the role of the real world. An agent is a graph of nodes that turns raw observations into actions, and it mirrors the robot's software stack 2301.04195 §3. A task adds the goal, reward and reset logic. A learning method can replace one node by cutting the agent's graph there, so the same world can host different tasks 2301.04195 §3. Many copies of the world are stepped in parallel on one GPU workstation 2301.04195 §5.5.

A simulator also needs shared rules for scoring what the agent does in it. For navigation, a working group proposes how to define success and distance, and which standard scenes to test on 1807.06757 §4 1807.06757 §6.

Many recent papers do not build a simulator. They train and score policies on an existing simulated benchmark, and the variants below cover both kinds of use. Simulation also serves as a closed-loop test bench for a vision model, and as a digital twin that a person teleoperates through.

Variants

Simulator frameworks and evaluation rules

World and agent split on Isaac Sim (Orbit)

The framework runs on NVIDIA Isaac Sim and bundles these parts 2301.04195 §3, §4, §5.1, §5.2:

• Scenes can be built in code, from scanned meshes, in the simulator's interface, or a mix 2301.04195 §3.
• Robots load USD assets. One interface covers a whole class of robots, such as legged robots 2301.04195 §3.
• Actuators. Joint commands pass through an actuator model before they set position, velocity or torque targets. The paper names DC-motor and series-elastic models, and names delays and torque limits as effects to model. Users identify their own actuator parameters 2301.04195 §3.
• Sensors. Physics sensors (range, force, contact) and rendering sensors (RGB, depth, normals) share one interface. Each sensor has its own timer and holds its last value between updates 2301.04195 §3.
• Objects and markers. Objects are passive, with textures and physical properties that can be randomized. Markers are primitive shapes drawn for debugging or to show goals 2301.04195 §3.
• Motion generators include GPU differential inverse kinematics, operational-space control and joint-level control, plus CPU planners for arms and mobile manipulators 2301.04195 §4.
• Teleoperation with keyboard, gamepad or SpaceMouse. The same path can record demonstrations 2301.04195 §4 2301.04195 §5.2.
• Learning wrappers for rl-games, RSL-rl and stable-baselines3, and demonstrations stored in the robomimic format 2301.04195 §5.1 2301.04195 §5.2.
• Benchmark tasks. The paper lists 26 tasks: 11 rigid-object, 13 deformable-object and 2 locomotion. Robots, objects, observations and action spaces can be switched through configuration 2301.04195 §1.

Card: Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments.

Physics kept on the GPU for PPO training (Isaac Gym)

Isaac Gym is a GPU physics simulator built on NVIDIA PhysX, with a Python tensor API. State, observations, rewards and actions stay on the GPU during training, so no large copy goes back to the CPU at each step [arxiv:2108.10470 §2.1] 2108.10470 §2.3. The paper trains PPO through rl_games on thousands of environments in parallel, and its reported runs use one NVIDIA A100 2108.10470 §4. The authors report Ant at reward 3000 in about 20 s with 4096 environments, and Humanoid at reward 5000 in under 4 min 2108.10470 §6.1.1 2108.10470 §6.1.2. On the OpenAI-style Shadow Hand task, they report 20 consecutive successes in under 1 h, against about 30 h for OpenAI's CPU-cluster run 2108.10470 §6.4.1 [arxiv:2108.10470 fig. 15]. That comparison mixes hardware, algorithms and seed counts, so it is not a like-for-like speed test 2108.10470 §6.4.1.

The paper's CC BY licence covers the paper, not the simulator. Check the simulator's own terms before commercial use. The card for this paper lists the repository and its pitfalls. Card: Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning.

Indoor digital replicas for embodied agents (Replica)

Replica is a dataset of 18 indoor 3D reconstructions. Each scene has a dense mesh, 16-bit HDR textures, per-primitive semantic labels in 88 classes, and mirror and glass reflectors 1906.05797 §1 1906.05797 §4. The authors built the scenes with a custom RGB-D rig, a SLAM system, TSDF fusion and manual annotation 1906.05797 §3. The release includes a folder for AI Habitat, which the paper recommends for machine learning [arxiv:1906.05797 §4.1] 1906.05797 §4.

In the paper's comparison, Replica has higher geometry resolution than Matterport3D and higher colour resolution than ScanNet. Matterport3D has the higher colour value, and ScanNet the higher geometry value [arxiv:1906.05797 tab. 1]. The paper reports no learning benchmark. Its realism test is a side-by-side comparison with no reported numbers 1906.05797 fig. 2.

The paper is CC BY 4.0, but the scene data is released under the repository's research terms, which limit use to non-commercial research or education. Check those terms before any commercial use. Card: The Replica Dataset: A Digital Replica of Indoor Spaces.

Evaluation protocol for simulated navigation

This is not a simulator. It is a set of recommendations from a working group on how to score navigation agents in simulated 3D scenes. The paper runs no experiments of its own 1807.06757 §1. It recommends the following 1807.06757 §2–§6:

• Stated goal and sensors. Name the goal type (PointGoal, ObjectGoal or AreaGoal) and the sensors, including any perfect odometry or GPS assumption 1807.06757 §2.
• Stated exploration regime. Say whether the agent had no prior exploration, a pre-recorded tour of the test scene, or a time-limited exploration budget. The budget examples are 500 m, 1,000 m and 2,000 m 1807.06757 §3.
• An explicit DONE action. Success is judged at the moment the agent emits DONE, using the shortest path distance inside the scene rather than a straight-line distance 1807.06757 §4.
• A distance threshold. A PointGoal or ObjectGoal episode succeeds when the distance at DONE is below a threshold. The default is twice the agent's body width 1807.06757 §4.
• SPL as the primary score, with the auxiliary measures listed in 1807.06757 §4 and the formula in 1807.06757 eq. 1. In the paper's worked examples, 50 % success with optimal paths gives 0.5. Full success with paths twice the shortest also gives 0.5. Half success with paths twice the shortest gives 0.25 1807.06757 §4.
• Simulator practice. Use continuous state spaces and SI units, and ship a route to deploy a simulated agent on a physical robot. The paper gives ROS as an example 1807.06757 §5.
• Standard scenes. Use the published scenario sets for SUNCG, Matterport3D, AI2-THOR and Gibson 1807.06757 §6.

Card: On Evaluation of Embodied Navigation Agents.

Simulators named in a foundation-model survey (ManiSkill2, Isaac Sim, Isaac Gym)

This is a survey, not a simulator. It groups simulators by their support for soft bodies, multiple controllers and parallel environments [arxiv:2402.02385 §4.2]. It names ManiSkill2 for soft bodies through the material point method, and NVIDIA Isaac Sim and Isaac Gym 2402.02385 §4.2. It gives no numbers for any of them, and its simulator statements date from early 2024 2402.02385 §4.2. Card: A Survey on Robotics with Foundation Models: toward Embodied AI.

Simulation as a renderer for augmented robot data

Cross-painting new robots into real demonstrations (AugE-Toolkit)

Here the simulator does not train the policy. It renders a second robot into recorded real demonstrations. The pipeline has three stages per frame: segment the source arm, inpaint the background, and replay the trajectory on a target arm in simulation 2512.13100 §4.1 2512.13100 fig. 2. The segmentation fuses a learned mask with a simulation mask, and the target arm's base position is tuned in simulation until its reach error falls below 1 cm 2512.13100 §4.2 (1) 2512.13100 §4.2 (2). A new robot is added by registering its MJCF model 2512.13100 §4.2 (3).

In their Robosuite study, the authors report that augmentation makes the source robot more robust to lighting and occlusion perturbations. On a held-out arm, source data alone performs poorly, while the augmented-plus-source policies often rival a policy trained on the target's augmented data [arxiv:2512.13100 §5.1] 2512.13100 fig. 3 2512.13100 fig. 4. In the simulation study, one augmented target alone reaches 26 to 65 % across four robots, and adding the source data improves that by 9 to 19 % 2512.13100 §5.1 2512.13100 fig. 4. On the physical tests, the authors report average gains of 24 % for OpenVLA-OFT and 45 % for pi0. The paper does not state whether these are absolute points or relative changes 2512.13100 §6.2 2512.13100 fig. 5. Card: OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning.

The simulated replay is one design choice among several. The authors say that diffusion-based cross-painting lowers policy success by 27 to 30 % in their comparison 2512.13100 §6.2 2512.13100 fig. 6.

Software between the simulator and the robot

Publish-subscribe middleware (ROS 2)

ROS 2 is not a simulator. It is the robot middleware that a controller can run on in simulation and on hardware. Nodes exchange typed messages over topics, services and actions, and the transport underneath is DDS, which can be swapped 2211.07752 §3.3 2211.07752 §3.4. Nodes in one process can pass a pointer instead of going through the middleware [arxiv:2211.07752 §3.6]. Quality-of-service settings control reliability, durability and history 2211.07752 §3.6. Security uses DDS-Security for authentication, access control and encryption 2211.07752 §3.7.

The paper's benchmark uses one publisher and one subscriber on CycloneDDS. The mean latency inside one process is 0.08 to 0.09 ms for messages up to 60 kB. At 8 MB it is 1.20 ms, against 8.37 ms between processes [arxiv:2211.07752 fig. 3a]. Under 20 % packet loss, the first second delivered about 8 of 29 messages 2211.07752 fig. 3b. The text quotes 95th-percentile values, but the plot shows means, so use the plotted figures and label them as means 2211.07752 §3.6 2211.07752 fig. 3a. The paper does not state its QoS for these runs, nor the number of repetitions. Treat the numbers as single observations.

Its five case studies cover quadruped, marine, drone, lunar-rover and warehouse-fleet deployments. The authors call them principally qualitative 2211.07752 §4. Card: Robot Operating System 2: Design, Architecture, and Uses In The Wild.

Digital twin as a teleoperation interface (TELESIM)

TELESIM moves a cube in an Isaac Sim digital twin, plans arm motion with Riemannian Motion Policy while avoiding collisions, and sends joint states to the real arm over ROS 2 2410.18727 §III 2410.18727 §III-B. In a user study, 74 non-experts (37 in the UK and 37 in Japan) stacked three cubes into towers for ten minutes on three robot arms 2410.18727 §IV. The UR5e built a mean of 10.50 towers per user, against 3.25 for Baxter and 1.03 for the UR3 2410.18727 §V-A 2410.18727 fig. 6. It also had the lowest NASA-TLX mental workload 2410.18727 §V-B. The UR5e was tested only in Japan, and the other two arms only in the UK, so arm, gripper, controller and country cannot be separated 2410.18727 §IV 2410.18727 §VI. Card: Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan.

Evolving bodies in simulation

Co-evolved morphology and PPO controllers (DERL, MuJoCo)

Here the simulator is used to search over robot bodies as well as controllers. A body is a tree of limbs joined by motor-driven hinges, simulated in MuJoCo. Each new body is a mutated copy of a parent, and it learns its own PPO controller from random weights for 5 million agent-environment steps. Its fitness is the average reward at the end of that training 2102.02202 §DERL 2102.02202 app. A. The search runs as asynchronous tournaments of four bodies, with a population of 576 and a budget of 4000 searched bodies 2102.02202 app. A 2102.02202 §DERL.

The authors report that bodies evolved on variable terrain, or with a manipulation task, learn eight test tasks faster and to higher reward than bodies evolved on flat ground 2102.02202 §Environmental complexity 2102.02202 fig. 4. Learning time to a fitness threshold falls by about half within 10 generations, although the selection rule does not reward learning speed 2102.02202 §Baldwin effect 2102.02202 fig. 5. The paper also links lower cost of work, a measure of energy use, to higher reward and faster learning 2102.02202 §Mechanistic underpinning 2102.02202 fig. 6.

The full run used 1152 CPUs and took under 16 hours, and the paper gives no single-machine estimate 2102.02202 §DERL 2102.02202 app. A. All results come from one simulator and are author-reported. Card: Embodied Intelligence via Learning and Evolution.

Simulated benchmarks for manipulation and locomotion policies

Each entry below names the benchmark its paper uses. The numbers are author-reported, and each comes from a different benchmark, so they cannot be compared with one another.

Frozen pretrained visual features on IsaacGym (PixMC)

A ViT-S masked autoencoder is pretrained on about 700k images of hands and objects, then frozen. A small PPO controller learns eight pixel-based tasks in the PixMC benchmark on IsaacGym 2203.06173 §4 2203.06173 §5. The authors report significantly higher success than a supervised ImageNet ViT-S on 7 of 8 tasks, and results close to an oracle policy that sees the true object state on 5 of 8 2203.06173 §5.1 2203.06173 fig. 5. Card: Masked Visual Pre-training for Motor Control.

Keyframe voxel behaviour cloning (RLBench)

The agent takes RGB-D input voxelised on a 100³ grid and a CLIP language goal, and predicts a translation voxel, rotation bins, gripper state and a collision flag at each keyframe 2209.05451 §3. The paper's results are all in RLBench simulation with a Franka Panda and four cameras 2209.05451 §4.1. The authors report that the agent beats a 3D U-Net baseline in 25 of 36 task-and-demo-count cells, with an average gain of 1.33x at 10 demos and 2.83x at 100 demos 2209.05451 §4.2 2209.05451 tab. 1. Card: Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation.

Multimodal prompts on VIMA-Bench (Ravens)

A single policy takes a prompt that interleaves text and object images, and it is trained by behaviour cloning on simulated expert data in VIMA-Bench, built on the Ravens simulator 2210.03094 §3 2210.03094 §5.1. The benchmark has four zero-shot levels, from placement changes (L1) to novel tasks (L4). At 200M parameters the authors report 81.5 % on L1 and 48.6 % on L4, against at most 58.0 % and 16.8 % for their re-implemented baselines 2210.03094 tab. 10. With 1 % of the imitation data, the policy reaches 36.3 % on L1, while the best baseline reaches 11.5 % 2210.03094 tab. 11. Card: VIMA: General Robot Manipulation with Multimodal Prompts.

Lifelong imitation on LIBERO

LIBERO is a simulated benchmark with 130 language-conditioned tasks in four suites, and each task has 50 demonstrations 2306.03310 §4.2 2306.03310 §4.4. The paper trains one policy on the tasks in a fixed order, using behaviour cloning, and scores forward transfer, negative backward transfer and AUC 2306.03310 §2.2 2306.03310 §5.1. The authors report that sequential fine-tuning has the best forward transfer on all four suites. They report that PackNet has the best AUC on LIBERO-Object and LIBERO-Goal, and that experience replay is robust across suites 2306.03310 §5.2 2306.03310 tab. 2. Card: LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning.

Video-model planning in simulated manipulation (UniPi)

A text-conditioned video diffusion model generates the frames of a task from the observed image and an instruction. An inverse-dynamics model then reads joint-space actions from those frames, and the actions run open-loop 2302.00111 §3. The method needs an offline dataset of observations, actions and a text description per episode 2302.00111 §2.2. On seen "put X in Y" instructions the authors report 59.1 % task completion, against at most 19.4 % for the best baseline 2302.00111 tab. 1. On three unseen multi-environment tasks, it reaches 75.5 % on Pack Object, against 21.7 % for the best baseline 2302.00111 tab. 3. Card: Learning Universal Policies via Text-Guided Video Generation.

LLM agents for multi-arm control in MuJoCo (RoCo)

Each robot is an LLM agent. The agents take turns in a dialog, and a proposed plan must pass ordered checks before a bidirectional RRT planner turns it into joint trajectories 2307.04738 §3.1 2307.04738 §3.2 2307.04738 §3.3. The paper runs zero-shot GPT-4 on six MuJoCo tasks with 20 runs each 2307.04738 §5.1. The authors' full dialog method reports success between 0.44 and 0.95 per task, and an oracle that plans for all robots at once has the higher mean 2307.04738 tab. 2. Averaged over the six tasks from that table (computed for this page, not reported by the authors), the oracle scores 0.81 and the dialog method 0.75. Card: RoCo: Dialectic Multi-Robot Collaboration with Large Language Models.

Event memory for VLA policies (RoboTwin-MeM, RMBench)

EventVLA keeps a bounded buffer of frames that a keyframe head flags as important, on top of a Qwen3-VL-4B backbone 2606.20092 §3. The authors report 75.2 % on RoboTwin-MeM, against 10.8 % for the best baseline in their table 2606.20092 tab. 2. On RMBench the average is 67.8 % [arxiv:2606.20092 tab. 1]. For tasks without a hidden cue, the paper reports 83.8 % on RoboTwin 2.0 Easy, but QwenFast scores 83.2 % on Hard against 81.6 % for EventVLA 2606.20092 tab. 3. Card: EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies.

Runtime critics and recovery around a frozen VLA (Zetta)

The VLA policy stays frozen. Code-based critics watch each rollout while it runs, and an orchestrator may approve a recovery routine when a critic fires 2608.16590 §2.1 2608.16590 eq. 3. An evolutionary agent rewrites the critics and recoveries from failed rollouts. A patch is kept only if it resolves the failure cluster and shows a gain on held-out seeds 2608.16590 §2.5 2608.16590 §2.6.2. The experiments use GR00T N1.5 on RoboCasa and π0.5 on LIBERO-Pro, both in simulation 2608.16590 §4.1.

The authors report that macro success on 18 RoboCasa tasks rises from 73.56 % to 93.56 % 2608.16590 tab. 2. On LIBERO-Pro, over 40 task-and-setting pairs, it rises from 32.00 % to 71.13 % 2608.16590 tab. 3. The abstract's 90.8 % LIBERO-Pro figure does not appear in any table, so this page does not use it 2608.16590 §1. Rollout throughput is 22.09 episodes per minute at concurrency 16, against 1.72 for the RPent baseline 2608.16590 §4.6.

The repository's licence could not be verified, so this page cites only the paper. Card: Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence.

Sim-to-real locomotion

Perceptive humanoid parkour (Unitree G1)

One RL policy maps forward depth images and proprioception to joint targets 2601.07718 §3.1. It trains in Isaac Sim and Isaac Lab with 4096 parallel environments on one RTX 4090 2601.07718 §3.7. Two training choices matter for transfer. An edge penalty discourages foot contact near sharp terrain edges 2601.07718 §3.3. Goals are placed on flat patches the robot can reach, instead of random velocity commands 2601.07718 §3.4. The authors report zero-shot hardware runs up to 2.5 m/s 2601.07718 §4.1.

On hardware, each terrain and gait pair was tried in 10 trials. With the edge penalty the authors report 10/10 on stair-up, against 8/10 without it, and 8/10 on the deep gap, against 3/10 2601.07718 tab. 1. The sample is small, so these counts are weak evidence. In simulation, removing the adversarial motion prior drops the small-box success to 0.00 % 2601.07718 tab. 2. Card: Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids.

Diffusion policy distilled from simulated walking (H1)

Here the simulator produces training data for a diffusion policy rather than scoring a policy. An AMP-trained walking policy runs in IsaacGym on a simulated Unitree H1 and writes the datasets 2411.01349 §3 2411.01349 §4. The authors build 24 datasets from eight randomization setups, each at 500K, 2M or 8M transitions 2411.01349 §3 2411.01349 §5. They report that terrain and perturbation randomization give the most robust policies 2411.01349 §5.1. The 8M "All" policy reaches success 1.0 on a flat, non-randomized target, with tracking 0.13 against 0.12 for the source policy 2411.01349 tab. 3. The authors conclude that locomotion needs far more data than manipulation 2411.01349 §6. The paper defers the diffusion settings to DiffuseLoco 2411.01349 §4. Training the 8M policy took about 23 h on one V100 2411.01349 §5. Card: The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control.

Simulated navigation and mobile manipulation

Three papers train a controller that moves a robot base, each in a different simulator. Only one of them has released code.

Base and arm moved together in ProcTHOR (HarmonicMM)

One PPO policy drives the base and the arm of a Stretch robot at the same time. Its inputs are two RGB cameras and a five-value arm reading. A pretrained DINOv2 ViT-S/14 encodes the images, and the released code keeps it frozen 2312.06639 §IV-C 2312.06639 app. Network Architecture. Stock ProcTHOR has no door or table-cleaning logic, so the authors added both to the simulator 2312.06639 §IV-A. Author-reported success averages 47 % over four simulated tasks, 17.6 points above an adapted ReLMoGen baseline 2312.06639 §V-B. On a real robot in one unseen apartment, without fine-tuning, the reported success rates are 66.6 % for cleaning, 60.0 % for pulling a door and 70.0 % for pushing one 2312.06639 tab. III. The released code uses several reward values that differ from the paper's, so state which set you use 2312.06639 app. Reward Function. Card: Harmonic Mobile Manipulation.

Final approach to a pose relative to an object (AMR)

The goal is a reference image with an object mask, a side to face, a distance and an angle. A transformer fuses frozen MAE tokens from RGB-D, 32 LiDAR bins and the goal, then predicts 12 base waypoints and a camera tilt 2411.14770 §III-B 2411.14770 §IV. The training data is 500k trajectories planned in Isaac Sim over HSSD scenes, and the policy is refined with DAgger 2411.14770 §III-A 2411.14770 §IV. Author-reported median error on unseen objects is 3 cm and 1 degree in simulation 2411.14770 §I. The completion rate is 95.9 % when the object is visible at the start, against 89.0 % for a FoundationPose baseline 2411.14770 fig. 6. That baseline has a CAD model and a perfect map, so the comparison is not like-for-like 2411.14770 §V-A. Card: Aim My Robot: Precision Local Navigation to Any Object.

Crop-row navigation with a learned turn rate (4WIS4WID)

A deep RL agent in Gazebo chooses only the turn rate. A fixed rule picks symmetric, zero-turn or lateral steering from the waypoint direction 2412.18865 §III-C 2412.18865 tab. I. Among PPO, SAC, TD3 and A2C, PPO gives the shortest episodes and the highest reward in the paper's runs 2412.18865 §IV-B 2412.18865 fig. 9. Author-reported success is 351 of 420 randomized multi-row trials, 83.57 % 2412.18865 tab. IV. The work is simulation only 2412.18865 §V. Card: Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning.

Simulation-grounded safety checks

Temporal safety rules over rollouts (SafeManip)

SafeManip checks recorded low-level rollouts against eight categories of temporal safety rules. The rules are written in LTLf and grounded in simulator state 2605.12386 §3.3 2605.12386 tab. 2. Each rule compiles to an automaton that steps through the trace 2605.12386 §3.4. The introduction calls this online, but the code and appendix run it over recorded traces 2605.12386 app. F. Violations are counted per triggered event, not per rollout 2605.12386 §4.2.

On LIBERO, the authors report that violations per triggered event correlate strongly with task success across seven policies (r = −0.99, p < 0.001). On RoboCasa365, with five policies, the correlation is positive and not significant 2605.12386 §5.1. The repository has no LICENSE file, so this page cites only the paper. Card: SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation.

Closed-loop testing of a vision model

Searching an arm cell for detector failures (MARTENS)

MARTENS places a YOLOv8 oriented-box detector and an arm controller in an Isaac Sim pick-and-place cell and runs them in a loop 2410.19277 §2.1 2410.19277 §3.1. A genetic algorithm searches box positions, rotations and light level for failing tests 2410.19277 §2.2. Against random search with the same budget, the authors report about 50 % more failures in use case 1 and 26 % more in use case 2 2410.19277 tab. 2 2410.19277 tab. 3. Fine-tuning the detector on the failing images repairs more than 99 % of them when they are re-run 2410.19277 §2.5 2410.19277 tab. 5. On about 100 real images per use case, the fine-tuned detectors reach a real-image mAP of 0.948 and 0.892, against 0.806 and 0.755 for detectors trained on real images only 2410.19277 §3.2.3 2410.19277 tab. 8. Each simulated test takes 280 s or 310 s on one A10G GPU 2410.19277 §3.1. Run one after another, a 220-test search takes about 17 to 19 hours (computed for this page). The paper names no code repository. Card: In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators.

When to use which

The first rows compare the reported behaviour of each source against the constraints an implementer will face. All figures are author-reported, and the last rows state what the evaluation papers ask for. None of the recommendations below has been tested by an independent study.

Implementation sketch

The agent-world loop from the paper. Sensors and nodes run at their own rates and hold their values between updates. Node order is not stated in the paper 2301.04195 §3.

# Advance the physics world by one step  [arxiv:2301.04195 §3]
world.simulate()

# Refresh each sensor only when its own timer is due  [arxiv:2301.04195 §3]
readings = {}
for sensor in world.sensors:
    if sensor.timer_due():
        sensor.refresh()
    readings[sensor.name] = sensor.last_value

# Pass data through the agent's nodes in order  [arxiv:2301.04195 §3]
values = readings
for node in agent.nodes:
    if node.timer_due():
        values = node.run(values)

# Send the final joint targets through the actuator model  [arxiv:2301.04195 §3]
return values

The real-robot bridge for the Franka arm, which the paper describes, upsamples 60 Hz joint commands to 1000 Hz with quintic segments between command knots. The paper does not give the coefficients or the boundary handling 2301.04195 §5.4.

Contradictions

Across papers, none known. The sources ask different questions, and they use different benchmarks, so their numbers do not measure the same thing. The GPU speed claims of Orbit and Isaac Gym are an example: Orbit reports on-par speed with IsaacGymEnvs 2301.04195 fig. 11, and Isaac Gym reports its own throughput on one A100 2108.10470 §5. The two sets of numbers use different tasks and hardware, so they do not conflict.

Within sources:

• Orbit: the abstract mentions 16 robotic platforms, and Section 4 lists platforms by type without a total 2301.04195 §4. The abstract says "more than 20" tasks, while Section 1 counts 26 2301.04195 §1.
• SafeManip: on LIBERO, more successful policies have fewer violations per triggered event (r = −0.99). On RoboCasa365 the correlation is positive, and it is not significant with five policies 2605.12386 §5.1.
• Zetta: the abstract and introduction give 90.8 % on LIBERO-Pro. The tables give 71.13 % as the 40-pair macro 2608.16590 §1 [arxiv:2608.16590 tab. 3]. The paper also gives different hardware in two places, 8 × RTX 4090 in §4.1 and 8 × A100 in §4.6 2608.16590 §4.1 2608.16590 §4.6.
• ROS 2: the text quotes 95th-percentile latencies, while the plotted values are means 2211.07752 §3.6 2211.07752 fig. 3a.
• MARTENS: the abstract gives 25 % to 50 % more failures, while the body gives 50 % and 26 % 2410.19277 tab. 2. The abstract also says fewer than 10 fine-tuning epochs, but the table gives 12 for the use-case-2 model 2410.19277 tab. 8.
• AMR: the inference rate is 12 Hz in the method section and 10 Hz in the conclusion 2411.14770 §IV 2411.14770 §VII.
• Diffusion humanoid: on the randomized target at 2M, the text says only perturbation training gives strong results, but the table shows 0.96 for terrain plus perturbation and 0.99 for "All" 2411.01349 §5.2 2411.01349 fig. 3.
• TELESIM: the NARS caption and the text read the scores in opposite directions, so check the sign before comparing with other studies 2410.18727 fig. 12 2410.18727 §V-C.
• DERL: the text says 7 test tasks for the main comparison, while the suite and the figure caption list 8 2102.02202 §Environmental complexity 2102.02202 fig. 4.

Papers

• Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments: a GPU-parallel robot-learning framework on Isaac Sim, with actuator and sensor models, benchmark tasks and a sim-to-real example.
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning: GPU-resident PhysX simulation with a tensor API, and PPO training on thousands of environments on one GPU.
• The Replica Dataset: A Digital Replica of Indoor Spaces: 18 indoor 3D scenes with HDR textures, reflectors and semantic labels, for embodied-agent research. No learning benchmark.
• On Evaluation of Embodied Navigation Agents: working-group recommendations for scoring simulated navigation agents, with SPL and standard scenario sets. It runs no experiments.
• Robot Operating System 2: Design, Architecture, and Uses In The Wild: the ROS 2 middleware design, a small latency benchmark, and five deployment case studies.
• Masked Visual Pre-training for Motor Control: a frozen masked-autoencoder encoder, pretrained on hand-object images, for PPO control in the PixMC benchmark.
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation: voxel-based keyframe behaviour cloning with language goals, evaluated in RLBench.
• VIMA: General Robot Manipulation with Multimodal Prompts: one policy for interleaved text-and-image prompts, with zero-shot levels in VIMA-Bench.
• Learning Universal Policies via Text-Guided Video Generation: planning with a text-conditioned video model and an inverse-dynamics model.
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning: a simulated lifelong imitation benchmark with 130 tasks and baseline results per algorithm and architecture.
• RoCo: Dialectic Multi-Robot Collaboration with Large Language Models: LLM agents for multi-arm planning, with checks and a joint motion planner in MuJoCo.
• Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids: a depth-based humanoid policy, trained in Isaac Sim and tested on hardware.
• SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation: temporal safety rules checked over simulated rollouts, with per-trigger violation rates.
• EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies: a VLA policy that stores visual keyframes in a bounded buffer, evaluated on RoboTwin and RMBench.
• Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence: code-based runtime critics and evolved recovery skills around a frozen VLA, evaluated on RoboCasa and LIBERO-Pro in simulation.
• Embodied Intelligence via Learning and Evolution: co-evolved MuJoCo bodies and PPO controllers, with asynchronous tournament selection and per-body lifetime learning.
• A Survey on Robotics with Foundation Models: toward Embodied AI: a survey of foundation models for robot planning and control, with a section on simulators and benchmarks. No new experiments.
• OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning: cross-paints new robot arms into existing demonstrations by simulated replay, with a Robosuite study and physical tests.
• Harmonic Mobile Manipulation: one PPO policy for base and arm in ProcTHOR door, table and fridge tasks, with one real-apartment test.
• Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan: a user study of 74 non-experts teleoperating three arms through a digital twin.
• In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators: a genetic search in an Isaac Sim pick-and-place cell for detector failures, followed by fine-tuning on them.
• The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control: diffusion policies trained on simulated RL walking data for a humanoid, across randomization setups and dataset sizes.
• Aim My Robot: Precision Local Navigation to Any Object: a transformer that drives a robot to a centimetre-level pose relative to an object, trained on simulated trajectories.
• Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning: PPO, SAC, TD3 and A2C in Gazebo for a four-wheel-steering farm robot between crop rows.