--- kind: concept slug: robot-simulation title: Robot simulation for learning summary: >- Training and testing robot policies in a simulated world before real hardware. Orbit and Isaac Gym run many physics copies in parallel on one GPU. Replica supplies indoor scenes, and many papers instead score policies on simulated benchmarks such as LIBERO, RoboTwin or RLBench. Choose a simulator by rendering needs, deformable objects and GPU budget. tags: [simulation, robot-learning, sim-to-real, evaluation, navigation, benchmark, runtime-monitoring, gpu-simulation, scene-dataset, middleware] papers: ["2301.04195", "1807.06757", "1906.05797", "2108.10470", "2203.06173", "2209.05451", "2210.03094", "2211.07752", "2302.00111", "2306.03310", "2307.04738", "2601.07718", "2605.12386", "2606.20092", "2608.16590", "2102.02202", "2402.02385", "2512.13100", "2312.06639", "2410.18727", "2410.19277", "2411.01349", "2411.14770", "2412.18865"] related: [imitation-learning, policy-gradient, learning-from-demonstration, simultaneous-localization-and-mapping, semantic-segmentation, continual-learning, masked-image-modeling, domain-randomization, sim-to-real-transfer, publish-subscribe-messaging, vision-language-action-model, llm-agent, temporal-logic-runtime-monitoring, neuroevolution, proximal-policy-optimization, data-augmentation, foundation-model, diffusion-models, teleoperation, digital-twin, metaheuristic-optimization, human-evaluation] status: draft updated_at: 2026-10-25 --- ## Overview Teaching a robot on hardware is slow and risky, so learning pipelines often train the policy in simulation first. A simulator has to balance fast and accurate physics, realistic sensors, deformable objects and easy task authoring. The designs on offer trade these off against each other [arxiv:2301.04195 §1]. The design the paper proposes splits the simulation in two. A **world** holds the robots, sensors, objects and markers, and plays the role of the real world. An **agent** is a graph of nodes that turns raw observations into actions, and it mirrors the robot's software stack [arxiv:2301.04195 §3]. A **task** adds the goal, reward and reset logic. A learning method can replace one node by cutting the agent's graph there, so the same world can host different tasks [arxiv:2301.04195 §3]. Many copies of the world are stepped in parallel on one GPU workstation [arxiv:2301.04195 §5.5]. A simulator also needs shared rules for scoring what the agent does in it. For navigation, a working group proposes how to define success and distance, and which standard scenes to test on [arxiv:1807.06757 §4] [arxiv:1807.06757 §6]. Many recent papers do not build a simulator. They train and score policies on an existing simulated benchmark, and the variants below cover both kinds of use. Simulation also serves as a closed-loop test bench for a vision model, and as a digital twin that a person teleoperates through. ## Variants ### Simulator frameworks and evaluation rules #### World and agent split on Isaac Sim (Orbit) The framework runs on NVIDIA Isaac Sim and bundles these parts [arxiv:2301.04195 §3, §4, §5.1, §5.2]: - **Scenes** can be built in code, from scanned meshes, in the simulator's interface, or a mix [arxiv:2301.04195 §3]. - **Robots** load USD assets. One interface covers a whole class of robots, such as legged robots [arxiv:2301.04195 §3]. - **Actuators.** Joint commands pass through an actuator model before they set position, velocity or torque targets. The paper names DC-motor and series-elastic models, and names delays and torque limits as effects to model. Users identify their own actuator parameters [arxiv:2301.04195 §3]. - **Sensors.** Physics sensors (range, force, contact) and rendering sensors (RGB, depth, normals) share one interface. Each sensor has its own timer and holds its last value between updates [arxiv:2301.04195 §3]. - **Objects and markers.** Objects are passive, with textures and physical properties that can be randomized. Markers are primitive shapes drawn for debugging or to show goals [arxiv:2301.04195 §3]. - **Motion generators** include GPU differential inverse kinematics, operational-space control and joint-level control, plus CPU planners for arms and mobile manipulators [arxiv:2301.04195 §4]. - **Teleoperation** with keyboard, gamepad or SpaceMouse. The same path can record demonstrations [arxiv:2301.04195 §4] [arxiv:2301.04195 §5.2]. - **Learning wrappers** for rl-games, RSL-rl and stable-baselines3, and demonstrations stored in the robomimic format [arxiv:2301.04195 §5.1] [arxiv:2301.04195 §5.2]. - **Benchmark tasks.** The paper lists 26 tasks: 11 rigid-object, 13 deformable-object and 2 locomotion. Robots, objects, observations and action spaces can be switched through configuration [arxiv:2301.04195 §1]. Card: [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](/papers/2301.04195). #### Physics kept on the GPU for PPO training (Isaac Gym) Isaac Gym is a GPU physics simulator built on NVIDIA PhysX, with a Python tensor API. State, observations, rewards and actions stay on the GPU during training, so no large copy goes back to the CPU at each step [arxiv:2108.10470 §2.1] [arxiv:2108.10470 §2.3]. The paper trains PPO through rl_games on thousands of environments in parallel, and its reported runs use one NVIDIA A100 [arxiv:2108.10470 §4]. The authors report Ant at reward 3000 in about 20 s with 4096 environments, and Humanoid at reward 5000 in under 4 min [arxiv:2108.10470 §6.1.1] [arxiv:2108.10470 §6.1.2]. On the OpenAI-style Shadow Hand task, they report 20 consecutive successes in under 1 h, against about 30 h for OpenAI's CPU-cluster run [arxiv:2108.10470 §6.4.1] [arxiv:2108.10470 fig. 15]. That comparison mixes hardware, algorithms and seed counts, so it is not a like-for-like speed test [arxiv:2108.10470 §6.4.1]. The paper's CC BY licence covers the paper, not the simulator. Check the simulator's own terms before commercial use. The card for this paper lists the repository and its pitfalls. Card: [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](/papers/2108.10470). #### Indoor digital replicas for embodied agents (Replica) Replica is a dataset of 18 indoor 3D reconstructions. Each scene has a dense mesh, 16-bit HDR textures, per-primitive semantic labels in 88 classes, and mirror and glass reflectors [arxiv:1906.05797 §1] [arxiv:1906.05797 §4]. The authors built the scenes with a custom RGB-D rig, a SLAM system, TSDF fusion and manual annotation [arxiv:1906.05797 §3]. The release includes a folder for AI Habitat, which the paper recommends for machine learning [arxiv:1906.05797 §4.1] [arxiv:1906.05797 §4]. In the paper's comparison, Replica has higher geometry resolution than Matterport3D and higher colour resolution than ScanNet. Matterport3D has the higher colour value, and ScanNet the higher geometry value [arxiv:1906.05797 tab. 1]. The paper reports no learning benchmark. Its realism test is a side-by-side comparison with no reported numbers [arxiv:1906.05797 fig. 2]. The paper is CC BY 4.0, but the scene data is released under the repository's research terms, which limit use to non-commercial research or education. Check those terms before any commercial use. Card: [The Replica Dataset: A Digital Replica of Indoor Spaces](/papers/1906.05797). #### Evaluation protocol for simulated navigation This is not a simulator. It is a set of recommendations from a working group on how to score navigation agents in simulated 3D scenes. The paper runs no experiments of its own [arxiv:1807.06757 §1]. It recommends the following [arxiv:1807.06757 §2–§6]: - **Stated goal and sensors.** Name the goal type (PointGoal, ObjectGoal or AreaGoal) and the sensors, including any perfect odometry or GPS assumption [arxiv:1807.06757 §2]. - **Stated exploration regime.** Say whether the agent had no prior exploration, a pre-recorded tour of the test scene, or a time-limited exploration budget. The budget examples are 500 m, 1,000 m and 2,000 m [arxiv:1807.06757 §3]. - **An explicit DONE action.** Success is judged at the moment the agent emits DONE, using the shortest path distance inside the scene rather than a straight-line distance [arxiv:1807.06757 §4]. - **A distance threshold.** A PointGoal or ObjectGoal episode succeeds when the distance at DONE is below a threshold. The default is twice the agent's body width [arxiv:1807.06757 §4]. - **SPL as the primary score**, with the auxiliary measures listed in [arxiv:1807.06757 §4] and the formula in [arxiv:1807.06757 eq. 1]. In the paper's worked examples, 50 % success with optimal paths gives 0.5. Full success with paths twice the shortest also gives 0.5. Half success with paths twice the shortest gives 0.25 [arxiv:1807.06757 §4]. - **Simulator practice.** Use continuous state spaces and SI units, and ship a route to deploy a simulated agent on a physical robot. The paper gives ROS as an example [arxiv:1807.06757 §5]. - **Standard scenes.** Use the published scenario sets for SUNCG, Matterport3D, AI2-THOR and Gibson [arxiv:1807.06757 §6]. Card: [On Evaluation of Embodied Navigation Agents](/papers/1807.06757). #### Simulators named in a foundation-model survey (ManiSkill2, Isaac Sim, Isaac Gym) This is a survey, not a simulator. It groups simulators by their support for soft bodies, multiple controllers and parallel environments [arxiv:2402.02385 §4.2]. It names ManiSkill2 for soft bodies through the material point method, and NVIDIA Isaac Sim and Isaac Gym [arxiv:2402.02385 §4.2]. It gives no numbers for any of them, and its simulator statements date from early 2024 [arxiv:2402.02385 §4.2]. Card: [A Survey on Robotics with Foundation Models: toward Embodied AI](/papers/2402.02385). ### Simulation as a renderer for augmented robot data #### Cross-painting new robots into real demonstrations (AugE-Toolkit) Here the simulator does not train the policy. It renders a second robot into recorded real demonstrations. The pipeline has three stages per frame: segment the source arm, inpaint the background, and replay the trajectory on a target arm in simulation [arxiv:2512.13100 §4.1] [arxiv:2512.13100 fig. 2]. The segmentation fuses a learned mask with a simulation mask, and the target arm's base position is tuned in simulation until its reach error falls below 1 cm [arxiv:2512.13100 §4.2 (1)] [arxiv:2512.13100 §4.2 (2)]. A new robot is added by registering its MJCF model [arxiv:2512.13100 §4.2 (3)]. In their Robosuite study, the authors report that augmentation makes the source robot more robust to lighting and occlusion perturbations. On a held-out arm, source data alone performs poorly, while the augmented-plus-source policies often rival a policy trained on the target's augmented data [arxiv:2512.13100 §5.1] [arxiv:2512.13100 fig. 3] [arxiv:2512.13100 fig. 4]. In the simulation study, one augmented target alone reaches 26 to 65 % across four robots, and adding the source data improves that by 9 to 19 % [arxiv:2512.13100 §5.1] [arxiv:2512.13100 fig. 4]. On the physical tests, the authors report average gains of 24 % for OpenVLA-OFT and 45 % for pi0. The paper does not state whether these are absolute points or relative changes [arxiv:2512.13100 §6.2] [arxiv:2512.13100 fig. 5]. Card: [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](/papers/2512.13100). The simulated replay is one design choice among several. The authors say that diffusion-based cross-painting lowers policy success by 27 to 30 % in their comparison [arxiv:2512.13100 §6.2] [arxiv:2512.13100 fig. 6]. ### Software between the simulator and the robot #### Publish-subscribe middleware (ROS 2) ROS 2 is not a simulator. It is the robot middleware that a controller can run on in simulation and on hardware. Nodes exchange typed messages over topics, services and actions, and the transport underneath is DDS, which can be swapped [arxiv:2211.07752 §3.3] [arxiv:2211.07752 §3.4]. Nodes in one process can pass a pointer instead of going through the middleware [arxiv:2211.07752 §3.6]. Quality-of-service settings control reliability, durability and history [arxiv:2211.07752 §3.6]. Security uses DDS-Security for authentication, access control and encryption [arxiv:2211.07752 §3.7]. The paper's benchmark uses one publisher and one subscriber on CycloneDDS. The mean latency inside one process is 0.08 to 0.09 ms for messages up to 60 kB. At 8 MB it is 1.20 ms, against 8.37 ms between processes [arxiv:2211.07752 fig. 3a]. Under 20 % packet loss, the first second delivered about 8 of 29 messages [arxiv:2211.07752 fig. 3b]. The text quotes 95th-percentile values, but the plot shows means, so use the plotted figures and label them as means [arxiv:2211.07752 §3.6] [arxiv:2211.07752 fig. 3a]. The paper does not state its QoS for these runs, nor the number of repetitions. Treat the numbers as single observations. Its five case studies cover quadruped, marine, drone, lunar-rover and warehouse-fleet deployments. The authors call them principally qualitative [arxiv:2211.07752 §4]. Card: [Robot Operating System 2: Design, Architecture, and Uses In The Wild](/papers/2211.07752). #### Digital twin as a teleoperation interface (TELESIM) TELESIM moves a cube in an Isaac Sim digital twin, plans arm motion with Riemannian Motion Policy while avoiding collisions, and sends joint states to the real arm over ROS 2 [arxiv:2410.18727 §III] [arxiv:2410.18727 §III-B]. In a user study, 74 non-experts (37 in the UK and 37 in Japan) stacked three cubes into towers for ten minutes on three robot arms [arxiv:2410.18727 §IV]. The UR5e built a mean of 10.50 towers per user, against 3.25 for Baxter and 1.03 for the UR3 [arxiv:2410.18727 §V-A] [arxiv:2410.18727 fig. 6]. It also had the lowest NASA-TLX mental workload [arxiv:2410.18727 §V-B]. The UR5e was tested only in Japan, and the other two arms only in the UK, so arm, gripper, controller and country cannot be separated [arxiv:2410.18727 §IV] [arxiv:2410.18727 §VI]. Card: [Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan](/papers/2410.18727). ### Evolving bodies in simulation #### Co-evolved morphology and PPO controllers (DERL, MuJoCo) Here the simulator is used to search over robot bodies as well as controllers. A body is a tree of limbs joined by motor-driven hinges, simulated in MuJoCo. Each new body is a mutated copy of a parent, and it learns its own PPO controller from random weights for 5 million agent-environment steps. Its fitness is the average reward at the end of that training [arxiv:2102.02202 §DERL] [arxiv:2102.02202 app. A]. The search runs as asynchronous tournaments of four bodies, with a population of 576 and a budget of 4000 searched bodies [arxiv:2102.02202 app. A] [arxiv:2102.02202 §DERL]. The authors report that bodies evolved on variable terrain, or with a manipulation task, learn eight test tasks faster and to higher reward than bodies evolved on flat ground [arxiv:2102.02202 §Environmental complexity] [arxiv:2102.02202 fig. 4]. Learning time to a fitness threshold falls by about half within 10 generations, although the selection rule does not reward learning speed [arxiv:2102.02202 §Baldwin effect] [arxiv:2102.02202 fig. 5]. The paper also links lower cost of work, a measure of energy use, to higher reward and faster learning [arxiv:2102.02202 §Mechanistic underpinning] [arxiv:2102.02202 fig. 6]. The full run used 1152 CPUs and took under 16 hours, and the paper gives no single-machine estimate [arxiv:2102.02202 §DERL] [arxiv:2102.02202 app. A]. All results come from one simulator and are author-reported. Card: [Embodied Intelligence via Learning and Evolution](/papers/2102.02202). ### Simulated benchmarks for manipulation and locomotion policies Each entry below names the benchmark its paper uses. The numbers are author-reported, and each comes from a different benchmark, so they cannot be compared with one another. #### Frozen pretrained visual features on IsaacGym (PixMC) A ViT-S masked autoencoder is pretrained on about 700k images of hands and objects, then frozen. A small PPO controller learns eight pixel-based tasks in the PixMC benchmark on IsaacGym [arxiv:2203.06173 §4] [arxiv:2203.06173 §5]. The authors report significantly higher success than a supervised ImageNet ViT-S on 7 of 8 tasks, and results close to an oracle policy that sees the true object state on 5 of 8 [arxiv:2203.06173 §5.1] [arxiv:2203.06173 fig. 5]. Card: [Masked Visual Pre-training for Motor Control](/papers/2203.06173). #### Keyframe voxel behaviour cloning (RLBench) The agent takes RGB-D input voxelised on a 100³ grid and a CLIP language goal, and predicts a translation voxel, rotation bins, gripper state and a collision flag at each keyframe [arxiv:2209.05451 §3]. The paper's results are all in RLBench simulation with a Franka Panda and four cameras [arxiv:2209.05451 §4.1]. The authors report that the agent beats a 3D U-Net baseline in 25 of 36 task-and-demo-count cells, with an average gain of 1.33x at 10 demos and 2.83x at 100 demos [arxiv:2209.05451 §4.2] [arxiv:2209.05451 tab. 1]. Card: [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](/papers/2209.05451). #### Multimodal prompts on VIMA-Bench (Ravens) A single policy takes a prompt that interleaves text and object images, and it is trained by behaviour cloning on simulated expert data in VIMA-Bench, built on the Ravens simulator [arxiv:2210.03094 §3] [arxiv:2210.03094 §5.1]. The benchmark has four zero-shot levels, from placement changes (L1) to novel tasks (L4). At 200M parameters the authors report 81.5 % on L1 and 48.6 % on L4, against at most 58.0 % and 16.8 % for their re-implemented baselines [arxiv:2210.03094 tab. 10]. With 1 % of the imitation data, the policy reaches 36.3 % on L1, while the best baseline reaches 11.5 % [arxiv:2210.03094 tab. 11]. Card: [VIMA: General Robot Manipulation with Multimodal Prompts](/papers/2210.03094). #### Lifelong imitation on LIBERO LIBERO is a simulated benchmark with 130 language-conditioned tasks in four suites, and each task has 50 demonstrations [arxiv:2306.03310 §4.2] [arxiv:2306.03310 §4.4]. The paper trains one policy on the tasks in a fixed order, using behaviour cloning, and scores forward transfer, negative backward transfer and AUC [arxiv:2306.03310 §2.2] [arxiv:2306.03310 §5.1]. The authors report that sequential fine-tuning has the best forward transfer on all four suites. They report that PackNet has the best AUC on LIBERO-Object and LIBERO-Goal, and that experience replay is robust across suites [arxiv:2306.03310 §5.2] [arxiv:2306.03310 tab. 2]. Card: [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](/papers/2306.03310). #### Video-model planning in simulated manipulation (UniPi) A text-conditioned video diffusion model generates the frames of a task from the observed image and an instruction. An inverse-dynamics model then reads joint-space actions from those frames, and the actions run open-loop [arxiv:2302.00111 §3]. The method needs an offline dataset of observations, actions and a text description per episode [arxiv:2302.00111 §2.2]. On seen "put X in Y" instructions the authors report 59.1 % task completion, against at most 19.4 % for the best baseline [arxiv:2302.00111 tab. 1]. On three unseen multi-environment tasks, it reaches 75.5 % on Pack Object, against 21.7 % for the best baseline [arxiv:2302.00111 tab. 3]. Card: [Learning Universal Policies via Text-Guided Video Generation](/papers/2302.00111). #### LLM agents for multi-arm control in MuJoCo (RoCo) Each robot is an LLM agent. The agents take turns in a dialog, and a proposed plan must pass ordered checks before a bidirectional RRT planner turns it into joint trajectories [arxiv:2307.04738 §3.1] [arxiv:2307.04738 §3.2] [arxiv:2307.04738 §3.3]. The paper runs zero-shot GPT-4 on six MuJoCo tasks with 20 runs each [arxiv:2307.04738 §5.1]. The authors' full dialog method reports success between 0.44 and 0.95 per task, and an oracle that plans for all robots at once has the higher mean [arxiv:2307.04738 tab. 2]. Averaged over the six tasks from that table (computed for this page, not reported by the authors), the oracle scores 0.81 and the dialog method 0.75. Card: [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](/papers/2307.04738). #### Event memory for VLA policies (RoboTwin-MeM, RMBench) EventVLA keeps a bounded buffer of frames that a keyframe head flags as important, on top of a Qwen3-VL-4B backbone [arxiv:2606.20092 §3]. The authors report 75.2 % on RoboTwin-MeM, against 10.8 % for the best baseline in their table [arxiv:2606.20092 tab. 2]. On RMBench the average is 67.8 % [arxiv:2606.20092 tab. 1]. For tasks without a hidden cue, the paper reports 83.8 % on RoboTwin 2.0 Easy, but QwenFast scores 83.2 % on Hard against 81.6 % for EventVLA [arxiv:2606.20092 tab. 3]. Card: [EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies](/papers/2606.20092). #### Runtime critics and recovery around a frozen VLA (Zetta) The VLA policy stays frozen. Code-based critics watch each rollout while it runs, and an orchestrator may approve a recovery routine when a critic fires [arxiv:2608.16590 §2.1] [arxiv:2608.16590 eq. 3]. An evolutionary agent rewrites the critics and recoveries from failed rollouts. A patch is kept only if it resolves the failure cluster and shows a gain on held-out seeds [arxiv:2608.16590 §2.5] [arxiv:2608.16590 §2.6.2]. The experiments use GR00T N1.5 on RoboCasa and π0.5 on LIBERO-Pro, both in simulation [arxiv:2608.16590 §4.1]. The authors report that macro success on 18 RoboCasa tasks rises from 73.56 % to 93.56 % [arxiv:2608.16590 tab. 2]. On LIBERO-Pro, over 40 task-and-setting pairs, it rises from 32.00 % to 71.13 % [arxiv:2608.16590 tab. 3]. The abstract's 90.8 % LIBERO-Pro figure does not appear in any table, so this page does not use it [arxiv:2608.16590 §1]. Rollout throughput is 22.09 episodes per minute at concurrency 16, against 1.72 for the RPent baseline [arxiv:2608.16590 §4.6]. The repository's licence could not be verified, so this page cites only the paper. Card: [Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence](/papers/2608.16590). ### Sim-to-real locomotion #### Perceptive humanoid parkour (Unitree G1) One RL policy maps forward depth images and proprioception to joint targets [arxiv:2601.07718 §3.1]. It trains in Isaac Sim and Isaac Lab with 4096 parallel environments on one RTX 4090 [arxiv:2601.07718 §3.7]. Two training choices matter for transfer. An edge penalty discourages foot contact near sharp terrain edges [arxiv:2601.07718 §3.3]. Goals are placed on flat patches the robot can reach, instead of random velocity commands [arxiv:2601.07718 §3.4]. The authors report zero-shot hardware runs up to 2.5 m/s [arxiv:2601.07718 §4.1]. On hardware, each terrain and gait pair was tried in 10 trials. With the edge penalty the authors report 10/10 on stair-up, against 8/10 without it, and 8/10 on the deep gap, against 3/10 [arxiv:2601.07718 tab. 1]. The sample is small, so these counts are weak evidence. In simulation, removing the adversarial motion prior drops the small-box success to 0.00 % [arxiv:2601.07718 tab. 2]. Card: [Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids](/papers/2601.07718). #### Diffusion policy distilled from simulated walking (H1) Here the simulator produces training data for a diffusion policy rather than scoring a policy. An AMP-trained walking policy runs in IsaacGym on a simulated Unitree H1 and writes the datasets [arxiv:2411.01349 §3] [arxiv:2411.01349 §4]. The authors build 24 datasets from eight randomization setups, each at 500K, 2M or 8M transitions [arxiv:2411.01349 §3] [arxiv:2411.01349 §5]. They report that terrain and perturbation randomization give the most robust policies [arxiv:2411.01349 §5.1]. The 8M "All" policy reaches success 1.0 on a flat, non-randomized target, with tracking 0.13 against 0.12 for the source policy [arxiv:2411.01349 tab. 3]. The authors conclude that locomotion needs far more data than manipulation [arxiv:2411.01349 §6]. The paper defers the diffusion settings to DiffuseLoco [arxiv:2411.01349 §4]. Training the 8M policy took about 23 h on one V100 [arxiv:2411.01349 §5]. Card: [The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control](/papers/2411.01349). ### Simulated navigation and mobile manipulation Three papers train a controller that moves a robot base, each in a different simulator. Only one of them has released code. #### Base and arm moved together in ProcTHOR (HarmonicMM) One PPO policy drives the base and the arm of a Stretch robot at the same time. Its inputs are two RGB cameras and a five-value arm reading. A pretrained DINOv2 ViT-S/14 encodes the images, and the released code keeps it frozen [arxiv:2312.06639 §IV-C] [arxiv:2312.06639 app. Network Architecture]. Stock ProcTHOR has no door or table-cleaning logic, so the authors added both to the simulator [arxiv:2312.06639 §IV-A]. Author-reported success averages 47 % over four simulated tasks, 17.6 points above an adapted ReLMoGen baseline [arxiv:2312.06639 §V-B]. On a real robot in one unseen apartment, without fine-tuning, the reported success rates are 66.6 % for cleaning, 60.0 % for pulling a door and 70.0 % for pushing one [arxiv:2312.06639 tab. III]. The released code uses several reward values that differ from the paper's, so state which set you use [arxiv:2312.06639 app. Reward Function]. Card: [Harmonic Mobile Manipulation](/papers/2312.06639). #### Final approach to a pose relative to an object (AMR) The goal is a reference image with an object mask, a side to face, a distance and an angle. A transformer fuses frozen MAE tokens from RGB-D, 32 LiDAR bins and the goal, then predicts 12 base waypoints and a camera tilt [arxiv:2411.14770 §III-B] [arxiv:2411.14770 §IV]. The training data is 500k trajectories planned in Isaac Sim over HSSD scenes, and the policy is refined with DAgger [arxiv:2411.14770 §III-A] [arxiv:2411.14770 §IV]. Author-reported median error on unseen objects is 3 cm and 1 degree in simulation [arxiv:2411.14770 §I]. The completion rate is 95.9 % when the object is visible at the start, against 89.0 % for a FoundationPose baseline [arxiv:2411.14770 fig. 6]. That baseline has a CAD model and a perfect map, so the comparison is not like-for-like [arxiv:2411.14770 §V-A]. Card: [Aim My Robot: Precision Local Navigation to Any Object](/papers/2411.14770). #### Crop-row navigation with a learned turn rate (4WIS4WID) A deep RL agent in Gazebo chooses only the turn rate. A fixed rule picks symmetric, zero-turn or lateral steering from the waypoint direction [arxiv:2412.18865 §III-C] [arxiv:2412.18865 tab. I]. Among PPO, SAC, TD3 and A2C, PPO gives the shortest episodes and the highest reward in the paper's runs [arxiv:2412.18865 §IV-B] [arxiv:2412.18865 fig. 9]. Author-reported success is 351 of 420 randomized multi-row trials, 83.57 % [arxiv:2412.18865 tab. IV]. The work is simulation only [arxiv:2412.18865 §V]. Card: [Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865). ### Simulation-grounded safety checks #### Temporal safety rules over rollouts (SafeManip) SafeManip checks recorded low-level rollouts against eight categories of temporal safety rules. The rules are written in LTLf and grounded in simulator state [arxiv:2605.12386 §3.3] [arxiv:2605.12386 tab. 2]. Each rule compiles to an automaton that steps through the trace [arxiv:2605.12386 §3.4]. The introduction calls this online, but the code and appendix run it over recorded traces [arxiv:2605.12386 app. F]. Violations are counted per triggered event, not per rollout [arxiv:2605.12386 §4.2]. On LIBERO, the authors report that violations per triggered event correlate strongly with task success across seven policies (r = −0.99, p < 0.001). On RoboCasa365, with five policies, the correlation is positive and not significant [arxiv:2605.12386 §5.1]. The repository has no LICENSE file, so this page cites only the paper. Card: [SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation](/papers/2605.12386). ### Closed-loop testing of a vision model #### Searching an arm cell for detector failures (MARTENS) MARTENS places a YOLOv8 oriented-box detector and an arm controller in an Isaac Sim pick-and-place cell and runs them in a loop [arxiv:2410.19277 §2.1] [arxiv:2410.19277 §3.1]. A genetic algorithm searches box positions, rotations and light level for failing tests [arxiv:2410.19277 §2.2]. Against random search with the same budget, the authors report about 50 % more failures in use case 1 and 26 % more in use case 2 [arxiv:2410.19277 tab. 2] [arxiv:2410.19277 tab. 3]. Fine-tuning the detector on the failing images repairs more than 99 % of them when they are re-run [arxiv:2410.19277 §2.5] [arxiv:2410.19277 tab. 5]. On about 100 real images per use case, the fine-tuned detectors reach a real-image mAP of 0.948 and 0.892, against 0.806 and 0.755 for detectors trained on real images only [arxiv:2410.19277 §3.2.3] [arxiv:2410.19277 tab. 8]. Each simulated test takes 280 s or 310 s on one A10G GPU [arxiv:2410.19277 §3.1]. Run one after another, a 220-test search takes about 17 to 19 hours (computed for this page). The paper names no code repository. Card: [In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators](/papers/2410.19277). ## When to use which The first rows compare the reported behaviour of each source against the constraints an implementer will face. All figures are author-reported, and the last rows state what the evaluation papers ask for. None of the recommendations below has been tested by an independent study. | Situation | Use | Why | |---|---|---| | Many rigid-body environments for RL on one GPU | Orbit's GPU pipeline | About 10x the throughput of CPU-vectorized frameworks, read from plots [arxiv:2301.04195 §5.5] [arxiv:2301.04195 fig. 11]. Against IsaacGymEnvs the paper reports on-par speed, because both use the same physics engine [arxiv:2301.04195 fig. 11]. | | Rigid-body PPO on one GPU with a tensor API and rl_games | Isaac Gym, or Orbit's GPU pipeline | Both keep physics on the GPU. Isaac Gym reports Ant at reward 3000 in about 20 s on one A100 [arxiv:2108.10470 §6.1.1]. Orbit reports on-par speed with IsaacGymEnvs, since both use the same physics engine [arxiv:2301.04195 fig. 11]. Choose by the API and stack you need. | | Deformable objects such as cloth | Orbit with the PBD solver | About 3x the throughput of DEDO in the cloth-hanging test [arxiv:2301.04195 fig. 12]. Finer meshes are more accurate but slower, and the largest run ran out of memory [arxiv:2301.04195 fig. 12]. | | Vision-based training with several cameras | Plan around rendering cost | Physics reaches up to 125,000 FPS, but ten 640 × 480 cameras reach 270 FPS [arxiv:2301.04195 §7]. Rendering, not physics, will limit throughput. | | Sim-to-real locomotion or manipulation on hardware | Use the paper as a pattern, with care | The paper deployed a locomotion policy on ANYmal and teleoperation and waypoint tasks on a Franka arm, and reports no success rates for either [arxiv:2301.04195 §5.4] [arxiv:2301.04195 fig. 8] [arxiv:2301.04195 fig. 9]. | | Imitation learning where the start state may change | Compare BC and BC-RNN in your own setup | On Franka-LiftCube with 100 trials per cell, BC-RNN keeps a success rate of 0.88 with a changed initial state, against 0.47 for BC [arxiv:2301.04195 tab. 2]. See [imitation-learning](/concepts/imitation-learning). | | Small machines or few GPUs | Check hardware first | The CPU-vectorized baselines reportedly crash at around 200–300 environments because of memory, and the throughput basis is a single RTX 3090 workstation [arxiv:2301.04195 §5.5]. Results may not carry over to smaller setups. | | Commercial product with a proprietary simulator runtime | Check licences before choosing | Isaac Sim is a separate dependency with its own licence. See the card's Pitfalls before any commercial use [arxiv:2301.04195 §6 fn.]. | | Pixel observations, frozen pretrained encoder, PPO | Compare against a supervised encoder and a random frozen encoder | Against a supervised ImageNet ViT-S, the frozen masked-autoencoder features win on 7 of 8 PixMC tasks [arxiv:2203.06173 §5.1]. Random frozen encoders reach zero or near zero on 6 of 8 tasks [arxiv:2203.06173 fig. 9]. In the authors' test, unfreezing the encoder gave zero success on Franka pick, for both random and MAE initialisation [arxiv:2203.06173 §5.3]. | | Few demonstrations for 6-DoF pick-and-place with calibrated RGB-D cameras | Voxel-based behaviour cloning, compared against image-based and voxel baselines | At 10 demos, the authors report 1.33x the baseline's average success. At 100 demos, 2.83x [arxiv:2209.05451 tab. 1]. The setup assumes a known workspace and calibrated cameras [arxiv:2209.05451 §3.2] [arxiv:2209.05451 app. B]. | | Language or image goals, with novel objects or tasks | Multimodal prompt policy, compared at the same data fraction | With 1 % of the data, the authors report 36.3 % on L1, against at most 11.5 % for the baselines [arxiv:2210.03094 tab. 11]. Adding one distractor object drops L3 from 78.7 % to 72.9 % [arxiv:2210.03094 app. E.5]. | | Skills learned in sequence, with forgetting measured | Experience replay or PackNet, not plain finetuning | Sequential finetuning gives the best forward transfer, but PackNet has the best AUC on LIBERO-Object and LIBERO-Goal, and experience replay is robust across suites [arxiv:2306.03310 tab. 2]. See [continual-learning](/concepts/continual-learning). | | Offline video and text data are available, and you want a plan a person can inspect | Video-model planner with an inverse-dynamics model | The paper needs episodes with video and text [arxiv:2302.00111 §2.2]. On real Bridge data, pretraining on internet video raised a classifier-scored success rate from 72.6 % to 77.1 %. This is a video-quality score, not a robot run [arxiv:2302.00111 tab. 4]. | | Multi-robot tasks with LLM agents and no task training | Dialog with environment feedback | On Pack Grocery, removing environment feedback lowers the success rate from 0.44 to 0.35, and lengthens successful runs from 9.9 to 18.0 steps [arxiv:2307.04738 tab. 2]. This is one task, so it does not show the effect on the others. | | The task needs a briefly visible cue, such as one hidden under a cover | Event memory for VLA policies | Averaged over RoboTwin-MeM, the authors report 75.2 % against 10.8 % for the best baseline [arxiv:2606.20092 tab. 2]. On RoboTwin 2.0 tasks without memory needs, the paper says memory does not hurt reactive control, and reports 81.6 % on Hard against 83.2 % for QwenFast [arxiv:2606.20092 tab. 3]. | | Humanoid must walk over stairs, gaps or rough terrain with a depth camera | Edge penalty and flat-patch goals in RL | On hardware, the edge penalty raises success on the deep gap from 3/10 to 8/10 [arxiv:2601.07718 tab. 1]. Each pair had 10 trials, so treat the gaps as indicative. | | Frozen robot policy that fails mid-episode, with many seeded simulated rollouts to learn from | Runtime critics with evolved recovery skills, gated on same-seed and held-out checks | Author-reported gains on RoboCasa and LIBERO-Pro, from 73.56 % to 93.56 % and from 32.00 % to 71.13 % [arxiv:2608.16590 tab. 2] [arxiv:2608.16590 tab. 3]. The method needs a simulator that can replay a rollout from its seed [arxiv:2608.16590 §4.1]. The LIBERO-Pro evaluation uses 20 episodes per entry, so each entry moves in steps of 5 points [arxiv:2608.16590 §4.4]. | | Need to judge safety, not just success, in simulated rollouts | Per-trigger violation rates from recorded rollouts | Success and safety can rank policies differently. On RoboCasa365 the most capable policies are often the most violation-prone per opportunity [arxiv:2605.12386 §5.1]. Compare policies on per-trigger rates, not per-rollout counts [arxiv:2605.12386 §4.2]. | | Comparing navigation results across papers or agents | Report under the protocol, with goal type, sensors and exploration regime stated | The paper asks for these statements because results cannot be compared without them [arxiv:1807.06757 §2] [arxiv:1807.06757 §3]. | | Agent that stops near the goal but never signals arrival | Add a DONE action before scoring | Success is judged only at DONE, so such an episode scores zero [arxiv:1807.06757 §4]. | | Comparing agents with different bodies or movement modes | Fix the distance definition for each comparison | Geodesic distances can differ between a ground vehicle and a drone. The paper's default is computed at a fixed height above the floor [arxiv:1807.06757 §6]. | | Choosing one headline number | SPL, with the auxiliary measures | The paper names SPL as primary and calls it strict. It expects an SPL of about 0.5 to be good in complex, unseen environments, and suggests calibrating against human SPL [arxiv:1807.06757 §4]. | | Need a test set that others can run | The standard scenario sets | The paper gives the scenario sets for four datasets, with split sizes and start-goal rules [arxiv:1807.06757 §6]. | | Existing real demonstrations cover few robot arms, and you need the same task on other arms | Cross-painting with simulated replay of the target arm | Author-reported gains on unseen arms and grippers, from 24 % (OpenVLA-OFT) and 45 % (pi0) on physical tests [arxiv:2512.13100 §6.2]. The augmentation needs known frame transforms, similar grippers, and a MJCF or URDF model for each target [arxiv:2512.13100 §3] [arxiv:2512.13100 §4.2 (3)]. | | Indoor scenes with reflectors and HDR textures for embodied-agent tests | Replica, if its research terms fit the project | The release has 18 scenes with dense meshes, HDR textures, reflectors and labels [arxiv:1906.05797 §4]. The paper reports no learning benchmark, so it does not show that policies trained on Replica transfer to real images [arxiv:1906.05797 §1]. | | Simulated controller that must run as several processes on a robot | ROS 2, with nodes composed into one process where latency matters | Intra-process messages are much faster than inter-process at 8 MB, 1.20 ms against 8.37 ms (mean, plotted) [arxiv:2211.07752 fig. 3a]. Default DDS is weak above about 1 MB, and tuning or composition helps [arxiv:2211.07752 §3.6]. | | Body design is open, and each candidate body needs its own learned controller, with a large CPU budget | Co-evolution of morphology and PPO controller in MuJoCo | Bodies evolved in harder terrain learn new tasks faster, according to the authors' figures [arxiv:2102.02202 fig. 4] [arxiv:2102.02202 fig. 5]. The full run used 1152 CPUs for under 16 hours [arxiv:2102.02202 §DERL]. Each candidate trains from random weights, so the cost per candidate is high [arxiv:2102.02202 app. A]. | | Base and arm must move together, e.g. to open a door while driving | Base-and-arm PPO policy, as in HarmonicMM | Author-reported 47 % average success over four simulated tasks [arxiv:2312.06639 §V-B]. The simulator needs door and table logic that stock ProcTHOR lacks [arxiv:2312.06639 §IV-A]. | | Final approach to a chosen side of an object, with no CAD model or map at run time | Goal-image navigation policy, as in AMR | Median error of 3 cm and 1 degree on unseen objects in simulation [arxiv:2411.14770 §I]. Training needs 500k planned trajectories and 150k steps on 8 A100 GPUs [arxiv:2411.14770 §IV]. Errors are measured only on runs that finish within 1 m [arxiv:2411.14770 §V]. | | Wheeled rover with unusual steering, where simulation-only evidence is acceptable | PPO with a fixed steering-mode switch, in Gazebo | 83.57 % success on randomized simulated trials [arxiv:2412.18865 tab. IV]. The paper has no real-robot tests [arxiv:2412.18865 §V]. | | Detector and controller must be tested together before deployment, and hours per simulated test are affordable | Closed-loop search for failures, then fine-tune on them | About 50 % more failures than random search in use case 1 and 26 % in use case 2 [arxiv:2410.19277 tab. 2] [arxiv:2410.19277 tab. 3]. The real-image gains rest on about 100 images per use case, so treat them as indicative [arxiv:2410.19277 §3.2.3]. | | Only simulated data for a humanoid walking policy | Diffusion policy on randomized RL data, from the largest (8M) dataset | 500K transitions did not give stable walking on the flat target [arxiv:2411.01349 §5.1]. The 8M set reached success 1.0 [arxiv:2411.01349 tab. 3]. Expect about 23 h on one V100 for that set [arxiv:2411.01349 §5]. | | Comparing teleoperation setups with non-experts | Digital-twin teleoperation study, changing one factor at a time | In this study arm, gripper, controller and country were confounded [arxiv:2410.18727 §IV] [arxiv:2410.18727 §VI]. Vary one factor at a time if you want to attribute a result to one cause. | ## Implementation sketch The agent-world loop from the paper. Sensors and nodes run at their own rates and hold their values between updates. Node order is not stated in the paper [arxiv:2301.04195 §3]. ```pseudocode # Advance the physics world by one step [arxiv:2301.04195 §3] world.simulate() # Refresh each sensor only when its own timer is due [arxiv:2301.04195 §3] readings = {} for sensor in world.sensors: if sensor.timer_due(): sensor.refresh() readings[sensor.name] = sensor.last_value # Pass data through the agent's nodes in order [arxiv:2301.04195 §3] values = readings for node in agent.nodes: if node.timer_due(): values = node.run(values) # Send the final joint targets through the actuator model [arxiv:2301.04195 §3] return values ``` The real-robot bridge for the Franka arm, which the paper describes, upsamples 60 Hz joint commands to 1000 Hz with quintic segments between command knots. The paper does not give the coefficients or the boundary handling [arxiv:2301.04195 §5.4]. ## Contradictions Across papers, none known. The sources ask different questions, and they use different benchmarks, so their numbers do not measure the same thing. The GPU speed claims of Orbit and Isaac Gym are an example: Orbit reports on-par speed with IsaacGymEnvs [arxiv:2301.04195 fig. 11], and Isaac Gym reports its own throughput on one A100 [arxiv:2108.10470 §5]. The two sets of numbers use different tasks and hardware, so they do not conflict. Within sources: - Orbit: the abstract mentions 16 robotic platforms, and Section 4 lists platforms by type without a total [arxiv:2301.04195 §4]. The abstract says "more than 20" tasks, while Section 1 counts 26 [arxiv:2301.04195 §1]. - SafeManip: on LIBERO, more successful policies have fewer violations per triggered event (r = −0.99). On RoboCasa365 the correlation is positive, and it is not significant with five policies [arxiv:2605.12386 §5.1]. - Zetta: the abstract and introduction give 90.8 % on LIBERO-Pro. The tables give 71.13 % as the 40-pair macro [arxiv:2608.16590 §1] [arxiv:2608.16590 tab. 3]. The paper also gives different hardware in two places, 8 × RTX 4090 in §4.1 and 8 × A100 in §4.6 [arxiv:2608.16590 §4.1] [arxiv:2608.16590 §4.6]. - ROS 2: the text quotes 95th-percentile latencies, while the plotted values are means [arxiv:2211.07752 §3.6] [arxiv:2211.07752 fig. 3a]. - MARTENS: the abstract gives 25 % to 50 % more failures, while the body gives 50 % and 26 % [arxiv:2410.19277 tab. 2]. The abstract also says fewer than 10 fine-tuning epochs, but the table gives 12 for the use-case-2 model [arxiv:2410.19277 tab. 8]. - AMR: the inference rate is 12 Hz in the method section and 10 Hz in the conclusion [arxiv:2411.14770 §IV] [arxiv:2411.14770 §VII]. - Diffusion humanoid: on the randomized target at 2M, the text says only perturbation training gives strong results, but the table shows 0.96 for terrain plus perturbation and 0.99 for "All" [arxiv:2411.01349 §5.2] [arxiv:2411.01349 fig. 3]. - TELESIM: the NARS caption and the text read the scores in opposite directions, so check the sign before comparing with other studies [arxiv:2410.18727 fig. 12] [arxiv:2410.18727 §V-C]. - DERL: the text says 7 test tasks for the main comparison, while the suite and the figure caption list 8 [arxiv:2102.02202 §Environmental complexity] [arxiv:2102.02202 fig. 4]. ## Papers - [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](/papers/2301.04195): a GPU-parallel robot-learning framework on Isaac Sim, with actuator and sensor models, benchmark tasks and a sim-to-real example. - [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](/papers/2108.10470): GPU-resident PhysX simulation with a tensor API, and PPO training on thousands of environments on one GPU. - [The Replica Dataset: A Digital Replica of Indoor Spaces](/papers/1906.05797): 18 indoor 3D scenes with HDR textures, reflectors and semantic labels, for embodied-agent research. No learning benchmark. - [On Evaluation of Embodied Navigation Agents](/papers/1807.06757): working-group recommendations for scoring simulated navigation agents, with SPL and standard scenario sets. It runs no experiments. - [Robot Operating System 2: Design, Architecture, and Uses In The Wild](/papers/2211.07752): the ROS 2 middleware design, a small latency benchmark, and five deployment case studies. - [Masked Visual Pre-training for Motor Control](/papers/2203.06173): a frozen masked-autoencoder encoder, pretrained on hand-object images, for PPO control in the PixMC benchmark. - [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](/papers/2209.05451): voxel-based keyframe behaviour cloning with language goals, evaluated in RLBench. - [VIMA: General Robot Manipulation with Multimodal Prompts](/papers/2210.03094): one policy for interleaved text-and-image prompts, with zero-shot levels in VIMA-Bench. - [Learning Universal Policies via Text-Guided Video Generation](/papers/2302.00111): planning with a text-conditioned video model and an inverse-dynamics model. - [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](/papers/2306.03310): a simulated lifelong imitation benchmark with 130 tasks and baseline results per algorithm and architecture. - [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](/papers/2307.04738): LLM agents for multi-arm planning, with checks and a joint motion planner in MuJoCo. - [Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids](/papers/2601.07718): a depth-based humanoid policy, trained in Isaac Sim and tested on hardware. - [SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation](/papers/2605.12386): temporal safety rules checked over simulated rollouts, with per-trigger violation rates. - [EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies](/papers/2606.20092): a VLA policy that stores visual keyframes in a bounded buffer, evaluated on RoboTwin and RMBench. - [Zetta ζ: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence](/papers/2608.16590): code-based runtime critics and evolved recovery skills around a frozen VLA, evaluated on RoboCasa and LIBERO-Pro in simulation. - [Embodied Intelligence via Learning and Evolution](/papers/2102.02202): co-evolved MuJoCo bodies and PPO controllers, with asynchronous tournament selection and per-body lifetime learning. - [A Survey on Robotics with Foundation Models: toward Embodied AI](/papers/2402.02385): a survey of foundation models for robot planning and control, with a section on simulators and benchmarks. No new experiments. - [OXE-AugE: A Large-Scale Robot Augmentation of OXE for Scaling Cross-Embodiment Policy Learning](/papers/2512.13100): cross-paints new robot arms into existing demonstrations by simulated replay, with a Robosuite study and physical tests. - [Harmonic Mobile Manipulation](/papers/2312.06639): one PPO policy for base and arm in ProcTHOR door, table and fridge tasks, with one real-apartment test. - [Breaking Down the Barriers: Investigating Non-Expert User Experiences in Robotic Teleoperation in UK and Japan](/papers/2410.18727): a user study of 74 non-experts teleoperating three arms through a digital twin. - [In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators](/papers/2410.19277): a genetic search in an Isaac Sim pick-and-place cell for detector failures, followed by fine-tuning on them. - [The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Control](/papers/2411.01349): diffusion policies trained on simulated RL walking data for a humanoid, across randomization setups and dataset sizes. - [Aim My Robot: Precision Local Navigation to Any Object](/papers/2411.14770): a transformer that drives a robot to a centimetre-level pose relative to an object, trained on simulated trajectories. - [Autonomous Navigation of 4WIS4WID Agricultural Field Mobile Robot using Deep Reinforcement Learning](/papers/2412.18865): PPO, SAC, TD3 and A2C in Gazebo for a four-wheel-steering farm robot between crop rows.