Sim-to-real transfer

Overview

A model trained on synthetic data has to work on inputs whose sensors, images, actuators and contacts differ from the simulator's. This wiki has twenty-four sources on the topic so far, each in its own setting. A humanoid policy is trained with reinforcement learning in Isaac Sim and Isaac Lab, then reports zero-shot runs on a Unitree G1 2601.07718 §4.1 2601.07718 app. B. A bipedal robot is trained to walk while carrying a moving load, with the load modelled in simulation, and its load policies reach lower top speeds on hardware than in simulation 2204.04340 §V 2204.04340 tab. 2. A full-size Digit walker is trained in simulation by distilling a state-based teacher into a student that sees only sensor history, and it walks outdoors with no falls over a week of full-day tests [arxiv:2303.03381 § Outdoor deployment]. Street-scene networks are pre-trained on rendered images and then fine-tuned on real ones 1810.08705 §5.1 1810.08705 tab. 4. A multimodal language model is fine-tuned on simulated objects, then run on a real arm, with adaptation after each real attempt 2312.16217 §4.1 2312.16217 §3.3. A GPU simulator is used for fast reinforcement learning, and its hand task randomises physical parameters during training. Its real-robot results come from a different robot, a TriFinger 2108.10470 §6.4.1 2108.10470 §6.4.2. An off-policy learning recipe trains a G1 locomotion policy in about 15 minutes on one consumer GPU, in simulation, and its hardware test covers whole-body tracking only 2512.01996 §3.1 2512.01996 §3.2. A grasp network is trained only on simulated grasps and rendered point clouds, then tested on a real Franka arm with one depth camera 2103.14127 §3.2 2103.14127 §4.3. A grasp detector trained on real depth scans is pretrained on simulated grasp exemplars first 1706.09911 §7.3. A humanoid loco-manipulation policy is split into a keypoint tracker and a task generator that reads depth, and it lifts, pushes and kicks objects on a Unitree G1, with qualitative hardware results only 2509.20322 §4.2. A second humanoid controller, for a Digit V3 trained in MuJoCo, randomises its dynamics and raises the difficulty of its torso pushes over a three-stage curriculum. It fails on 5% of 20 hardware motions, against 0.450% in simulation 2408.07295 §4.4 2408.07295 tab. 3 2408.07295 §5.6. The measures used to close the gap fall into a few groups: making simulated input look like real input, corrupting and delaying observations, correcting a contact effect the simulator gets wrong, randomising simulator parameters during training, training on the specific payload that will be carried, pre-training on synthetic data then fine-tuning on real data, distilling a state-based teacher into a sensor-only student [arxiv:2303.03381 § Teacher state-policy supervision] 2509.20322 §3.2, distilling a simulated height-map latent into a depth-camera encoder 2512.12993 §IV.C, and adapting at test time from real outcomes 2601.07718 §3.2 2601.07718 §3.3 2108.10470 app. A.4.1 1810.08705 §5.1 1706.09911 §7.3 2312.16217 §3.3. Noise added to simulated depth during training is one more such measure 2103.14127 fig. 6. A car-parking policy is trained in a Gaussian-splatting copy of scanned garages and tested on a real vehicle, with a critic that uses simulator-only maps 2605.08713 §4.2 2605.08713 tab. 4. A mobile manipulator with a learned arm and base is trained in simulated houses and run in one unseen real apartment with no further training 2312.06639 tab. III. A quadruped is trained in simulation with slippery, raised pipe entrances and random pushes, and then crosses real PVC pipes 2412.13621 §II-A4 2412.13621 tab. V. A wheeled robot copies a planner in simulation, and a geometric check guards its motion on hardware 2406.14947 §II-C 2406.14947 §IV-D. A vision detector is first stress-tested in a simulated work cell, and its failures are used to fine-tune it before real images are tried 2410.19277 §2.2 2410.19277 tab. 8. Surveys name the families of methods but give no transfer numbers 2406.01152 §5 2404.06229 §2.3. For the choice of simulator itself, see robot-simulation.

Variants

The sections below draw on twenty-four papers, and the citations name each source. The papers use different tasks, benchmarks and metrics, so no number on this page compares one paper's method with another's. The list shows the parts of each system. The two surveys are summarised in the last subsection.

Make simulated depth look like real depth

Two image pipelines map simulated and real depth into one perception space 2601.07718 §3.2 2601.07718 fig. 3. The simulated pipeline adds range-dependent Gaussian noise, masks contiguous pixels to imitate stereo matching failures, blurs to imitate motion blur, and with a fixed probability replaces the whole frame with noise 2601.07718 app. A.4. The real pipeline crops and resizes to the simulator's geometry, fills zero-valued pixels caused by occlusion or disparity shadows, and blurs to suppress sensor jitter 2601.07718 app. A.4. The policy sees several depth frames spaced a fixed number of steps apart, which covers a window of several steps 2601.07718 app. A.4 2601.07718 eq. 6.

Distil a height-map latent into a depth-camera student

A Digit walker reads a 16-value latent from a convolutional VAE that encodes a 20 by 40 height map, and the latent is learned in simulation 2512.12993 §II.B. For the real robot, a second encoder is trained on depth images from two cameras, on the torso and the pelvis. Its latent is pulled toward the height-map latent with a mean-squared loss, and the VAE loss stays as a regulariser 2512.12993 §IV.C. Simulated depth is post-processed to look like the sensor: cropping, blur, edge shadowing, clipping at 2 m, inpainting, pixel dropout and leg masking 2512.12993 §IV.C 2512.12993 fig. 12.

The sim-to-real evidence here is a comparison of simulated and real depth images. The paper reports no policy run on hardware, and lists hardware deployment as future work 2512.12993 fig. 12 2512.12993 §V. The policy results are simulation results: above 80% success on every tested terrain for the height-map policies, and one check on stairs in a second simulator with realistic dynamics 2512.12993 §IV.E 2512.12993 §IV.F.

Noise and delay on the observations

During training the actor receives each proprioceptive input with uniform noise. The ranges are ±0.2 rad/s on angular velocity, ±0.05 on projected gravity, ±0.01 rad on joint positions and ±0.5 rad/s on joint velocities 2601.07718 tab. 4. The critic sees noise-free states and the base linear velocity 2601.07718 tab. 3. Depth is delayed by one frame in training, to imitate sensor latency 2601.07718 §3.2 2601.07718 app. A.4. On hardware, depth is captured at 60 Hz and the policy runs at 50 Hz on the latest depth frame 2601.07718 §3.2 2601.07718 app. B.

Correct a contact effect the simulator gets wrong

Reinforcement learning agents tend to step on the nearest terrain edge, which saves energy. Contact in simulation is simplified, so this works there, but on hardware it causes slips 2601.07718 §3.3. The fix finds sharp edges, models each as a cylinder, and penalises foot points that penetrate a cylinder, scaled by the point's speed 2601.07718 §3.3 2601.07718 eq. 1. The weight is −4.0 2601.07718 tab. 5.

The effect shows on hardware, not in simulation. In simulation, the variant with the penalty and the variant without it differ by at most 0.13 points on stair up, stair down and deep gap 2601.07718 tab. 1. On the robot, the penalty variant reached 10/10, 10/10 and 8/10 on those three terrains, against 8/10, 6/10 and 3/10 without it 2601.07718 tab. 1. The paper does not state how the "no edge" variant was built 2601.07718 §4.2.

Staged gait policies and switching on the robot

Three specialised policies (stairs, platforms, running) are fine-tuned from a walking checkpoint, for 40,000 iterations each 2601.07718 app. A.8. On the robot, switching policies means stopping, standing, then starting the new policy 2601.07718 §4.1.

Randomise the simulator during training

The Shadow Hand OpenAI task resamples physical parameters at each reset: object dimensions, object and link masses, surface friction, joint damping, actuator P-gain, joint limits and gravity 2108.10470 tab. 18. Scaling factors are drawn uniformly or log-uniformly, and additive terms from normal distributions. Parameters are re-randomised on each reset, with at least 720 steps between resets 2108.10470 app. A.4.1.

The policy sees only observations that could be measured on a real hand. The value function also gets privileged simulator state, an asymmetric actor–critic 2108.10470 §6.4.1. The authors follow the OpenAI randomisation setup, with changes listed in the same section 2108.10470 §6.4.1. The reported result is in simulation: 20 consecutive successes in under 1 h on one A100, against about 30 h for OpenAI's CPU-cluster run 2108.10470 §6.4.1 2108.10470 fig. 15. The standard hand task reaches the same milestone in under 35 min without randomisation, but it uses different control rates and networks, so the two times cannot isolate the effect of randomisation 2108.10470 tab. 4 2108.10470 tab. 17.

The card records two hardware claims, and neither applies the randomisation settings above. A TriFinger real-robot task reports a 55% mean success rate, labelled sim-to-real 2108.10470 §6.4.2 2108.10470 fig. 17. An ANYmal rough-terrain policy, trained in under 20 min, is labelled sim-to-real transfer 2108.10470 §6.1.4 2108.10470 fig. 10. The card records no randomisation settings for either robot.

Train on simulated grasps with depth noise, then test on a real depth camera

A grasp network is trained with no real grasps. For each training scene, a virtual camera renders a point cloud, and a point counts as a positive contact when a valid simulated grasp lies within 5 mm of it 2103.14127 §3.2. Training with Gaussian noise helps generalisation, according to the paper's figure, which gives no noise level 2103.14127 fig. 6. On a Franka Panda with an Intel RealSense L515, the authors report 90.20% grasp success on 51 unseen objects in nine cluttered scenes. The best baseline, 6-DoF GraspNet with CollisionNet, scores 80.39% 2103.14127 tab. 1 2103.14127 §4.3. Card: Contact-GraspNet. The hardware test uses one robot and one depth camera, so it does not show transfer to other sensors 2103.14127 §4.3.

Distil a state-based teacher into a sensor-history student

The teacher is an MLP trained on the full simulator state, with hidden sizes [512, 512, 256, 128] 2303.03381 § Model architecture. The student is a causal transformer. It reads a window of 16 observation-action pairs, one token per pair, and outputs the next action 2303.03381 § Model architecture. Its loss adds an RL term to λ times a KL term against the teacher, and λ is annealed to zero, reaching zero near the middle of training 2303.03381 eq. 2 2303.03381 § Joint optimization with reinforcement learning. Both terms are optimised on-policy, with PPO as the RL algorithm [arxiv:2303.03381 § Policy learning].

Training randomises robot dynamics, control parameters, environment physics, observation noise and observation delay. Terrains are smooth planes, rough planes and smooth slopes 2303.03381 § Domain randomization. The randomisation table and the command ranges are in a supplement that is not in the arXiv source, so the card records no values [arxiv:2303.03381 § Domain randomization]. Isaac Gym cannot model Digit's closed leg chains, so the rods are replaced by a stiff virtual spring with sub-stepping. The stiffness is not stated 2303.03381 § Closed kinematic chain.

A vendor simulator is used as a filter before hardware. It does not change the network weights, and it only removes unsafe policies [arxiv:2303.03381 § Sim-to-real transfer]. The policy runs at 50 Hz on hardware, with joint PD control at 1 kHz 2303.03381 § Sim-to-real transfer.

The paper reports no results table. On hardware, the policy walks on rubbers, cloths, cables and bubble wrap at 0.15 m/s, and crosses slopes up to 8.7%, against 10% in training 2303.03381 § Rough terrain. In a simulator comparison, the policy is comparable to the vendor's controller on slopes and better on steps and unstable planks, with values only in a figure 2303.03381 fig. 2. Among student architectures, the transformer is best by a considerable margin, and a longer context helps on both tested slopes 2303.03381 fig. 8. The joint objective is best in the same figure 2303.03381 fig. 8. The paper gives the values of these comparisons only in figures.

Randomise dynamics and stage the torso pushes on a humanoid

The Digit V3 policy randomises joint damping, link masses, centre-of-mass positions, encoder noise and ground friction in every curriculum stage 2408.07295 §4.5. The paper gives no ranges for these, and refers to earlier Digit locomotion work for them, so the card records no values 2408.07295 §4.5.

The training curriculum has three stages 2408.07295 §4.4. Stage 1 uses single torso pushes of 80 to 800 N, each lasting one policy step, with a 1% chance per step. Stage 2 uses continuous pushes of 20 to 150 N, lasting 20 to 50 policy steps. Stage 3 does not state its perturbations 2408.07295 §4.4. The paper's no-curriculum variant fails on 4.875% of mimicry motions in simulation, against 0.450% for the full method 2408.07295 tab. 5. The two variants were not trained for the same number of steps, and the paper reports neither count 2408.07295 §5.5.

On hardware the same policy has an MEPE of 0.117 m, against 0.098 m in simulation 2408.07295 tab. 3. The Future Work section names wide foot placements and single-foot stances as sim-to-real gaps still open 2408.07295 §6.

Off-policy learning with massively parallel simulation

FastSAC and FastTD3 are off-policy learners. They reuse past transitions from a replay buffer, and they train on thousands of simulated robots at once 2512.01996 §2.1. The authors name a few settings that matter at this scale: action bounds set from each joint's limit and default position, layer normalisation, a critic target that averages the two Q-values, a distributional critic, and an initial entropy temperature of 0.001 2512.01996 §2.1 2512.01996 alg. 1. The reward has fewer than ten terms 2512.01996 §2.2.

For locomotion, the randomised quantities are pushes, action delay, PD gains, mass and friction, and centre of mass on the G1 only 2512.01996 §3.1. Pushes come every 5 to 10 s, and every 1 to 3 s in the strong variant 2512.01996 fig. 3.

The authors report velocity-tracking policies for the G1 and T1 learned in 15 minutes on one RTX 4090 under strong randomisation, and they say PPO trains more slowly. The numbers are in figures only 2512.01996 §3.1 2512.01996 fig. 3. Hardware is shown for G1 whole-body tracking, with a dance motion of over two minutes among others 2512.01996 §3.2 2512.01996 fig. 6. The locomotion curves are simulation results 2512.01996 §3.1. The released code differs from the paper in several settings, so the paper's recipe is not the code's default the card.

Train on the specific payload in simulation

A Cassie robot carries one of four loads: a box on a tray, a cart on ropes, a pole with a swinging weight at each end, or a water jug modelled as a weight on springs 2204.04340 §III-B. The policy sees only its own joints and body, and the load changes only the simulated robot 2204.04340 §III-B.

Training with the specific load recovers most of the lost performance. For the carrying pole, the pass rate goes from 0.0014 for the base walker to 0.9578 for the specific policy 2204.04340 tab. 2. A general policy trained on all five models is close to the specific ones, and on the tray box, cart and water jug it is higher 2204.04340 tab. 2. Its pole pass rate is 0.8342 2204.04340 tab. 2.

Starting training from the base walker gives pass rates within about two points of training from scratch on the tray box, and higher on the other three loads 2204.04340 tab. 3. The authors say bootstrapped policies need on the order of 100 million fewer samples to reach the same reward, which they read from a learning curve 2204.04340 §IV-C 2204.04340 fig. 4.

On hardware, the top speed in walk-in-place ramp tests is 3.0 m/s for the base walker, against 3.9 m/s in simulation. The tray box and carrying pole policies reach 1.0 and 0.8 m/s, against 2.1 and 2.5 m/s in simulation 2204.04340 §V 2204.04340 tab. 2. The paper expects dynamics randomisation alone to fall short with loads 2204.04340 §V 2204.04340 §VI. It does not describe how the loads were mounted on the hardware 2204.04340 §V.

Pre-train on synthetic data, then fine-tune on real data

Synscapes is a set of 25,000 path-traced street scenes with per-pixel labels, instance and depth maps 1810.08705 §3. Networks trained on it alone score well below a Cityscapes-trained baseline on real Cityscapes images: DeepLab v3+ reaches 50.35 mean IoU against 76.56, and FRRN 45.20 against 68.27 1810.08705 tab. 4. Trained on Synscapes and then fine-tuned on Cityscapes, the scores rise above the baseline, to 78.85 for DeepLab v3+ and 74.52 for FRRN 1810.08705 tab. 4. The two comparison synthetic sets also end above their baselines after fine-tuning, by less: 77.57 and 77.45 for DeepLab v3+ 1810.08705 tab. 4.

For detection, a Faster R-CNN trained only on Synscapes reaches mAP 0.092 on the KITTI validation set, against 0.456 when trained on KITTI. Training on both gives 0.519 1810.08705 tab. 6. The authors say the segmentation comparison alone does not prove a smaller domain shift 1810.08705 §4.1. They say the gap between the synthetic sets is most likely due to sensor simulation and feature variation 1810.08705 §4.2.

Grasp Pose Detection pretrains its classifier on 207,000 simulated grasp exemplars made from 400 CAD models 1706.09911 §7.3. On the real BigBird test set, the network trained on these alone reaches 83% accuracy, against about 90% for real training data 1706.09911 §7.3. Simulated pretraining followed by real fine-tuning reaches the same accuracy in 4,000 iterations, against 20,000 without pretraining 1706.09911 §7.3 1706.09911 fig. 9. The robot trials are not described as using this pretraining 1706.09911 §8.6. Card: Grasp Pose Detection.

The earlier grasp-detection paper uses the same two-stage idea. Its 15-channel classifier is trained only on 207,000 exemplars from 400 CAD models in 3DNET, for 30,000 iterations, and reaches 83% accuracy on a BigBird test set. Its best representation, trained on real BigBird data, reaches about 90% on the same set 1603.01564 §4.3. Starting from the 3DNET weights, the network reaches at 4,000 iterations the accuracy a random start reaches at 20,000, and the paper says the advantage shrinks as training goes on 1603.01564 §4.3. Card: High precision grasp pose detection in dense clutter.

For a box-detection model on a pick-and-place arm, a detector trained only on synthetic images scores 0.911 mAP on about 100 real UC-1 images and 0.723 on UC-2, with no training on real data 2410.19277 tab. 7. The real images are scored in full there, and in a separate 20 % test split later. Starting from the repaired synthetic model and training on 60 % of the real images gives 0.948 mAP on UC-1 and 0.892 on UC-2. Training from the standard pre-trained weights on real images alone gives 0.806 and 0.755 2410.19277 tab. 8. The fine-tuned models converge in 6 to 12 epochs, against 26 for real-only training 2410.19277 tab. 8. The UC-2 camera has a fisheye effect that the simulation does not model, and the authors blame it for the weaker UC-2 result 2410.19277 §3.2.3.

Adapt at test time from real trial outcomes

ManipLLM fine-tunes LLaMA-Adapter on simulated articulated objects, with four fine-tuning tasks that share one image 2312.16217 §3.1.2. It answers with a contact pixel and two gripper directions 2312.16217 §3.1.2. On simulated PartNet-Mobility objects, it reaches 0.54 initial-movement success on 10 unseen categories, against 0.41 for the best baseline, Implicit3D 2312.16217 tab. 1. On a real Franka arm with 7 categories and 5 trials each, the per-category counts sum to 28 of 35; the paper does not report that total 2312.16217 tab. 3 2312.16217 §4.4.

After a real attempt, the model is asked the affordance yes/no question about its predicted contact point, with the real outcome as the answer. Only the visual adapter is updated 2312.16217 §3.3. The paper shows a real example with and without this step only qualitatively 2312.16217 §4.4 2312.16217 fig. 5. After the initial contact is established, an active impedance loop pushes along several candidate directions and keeps the one that moves the end-effector most 2312.16217 §3.2. The number of candidates and the push sizes are not stated 2312.16217 §3.2. The public code contains no impedance or test-time adaptation code the card.

Train in a scanned Gaussian-splatting simulator with a privileged critic

REAP trains a camera-based Soft Actor-Critic parking policy in a 3D Gaussian Splatting simulator. The scene is built from handheld LiDAR and camera scans, and the LiDAR points initialise the Gaussians 2605.08713 §5.1. The actor sees only surround-view images. The critic also gets BEV maps from the simulator, which are used only in training 2605.08713 §4.2. Camera intrinsics and extrinsics are perturbed during training, and the target position is disturbed too 2605.08713 §4.4.1 2605.08713 §5.3.

The policy learns from a mix of two signals. A Reeds-Shepp planner supplies behaviour cloning, and the probability of using it falls linearly from 1 to 0 over 8,000 epochs 2605.08713 §4.2 2605.08713 §5.3. Removing the perception pre-training stage gives 50.5% simulated success, against 83.5% for the full method, and removing the hybrid RL stage gives 0.0% 2605.08713 tab. 2.

The simulator comparison in the paper gives 70.0% real parking success for 3DGS, against 45.0% for a mesh simulator and 0.0% for a CAD simulator built in CARLA. A 3DGS build takes about 1 h, and a CAD build more than 10 h 2605.08713 tab. 3. The five real-vehicle scenarios reach 55% to 85% success, with 20 trials each, on one vehicle with one seed 2605.08713 tab. 4. The paper does not claim generalisation to unseen outdoor scenes or to dynamic obstacles 2605.08713 §6.

Zero-shot mobile manipulation in simulated houses

One PPO policy drives a Stretch robot's base, arm height and extension, and wrist together, from two RGB cameras and five values of arm proprioception 2312.06639 §IV-C. It is trained in ProcTHOR houses. The authors added door-opening and table-cleaning logic to the simulator, because the stock version has none 2312.06639 §IV-A. The paper uses a pretrained DINOv2 backbone, and the released code keeps it frozen the card. Without the pretrained backbone, the pull task reaches 0% success in simulation 2312.06639 tab. IV.

On a real robot in one apartment with a new layout, and with no fine-tuning, the policy cleaned 8 of 12 tables, opened 9 of 15 doors by pulling and 7 of 10 by pushing 2312.06639 tab. III. The fridge was not run on hardware, because the magnetic seal was too strong for the robot to pull 2312.06639 §V-C. The paper does not state what share of a table counts as a successful clean the card.

Precise pose beside an object, with history on the robot

Aim My Robot is trained by behaviour cloning on 500,000 simulated trajectories in Isaac Sim, then refined with DAgger 2411.14770 §IV. Its trajectories come from a planner, not from human driving 2411.14770 §III-A. The network reads frozen MAE tokens for the current image and the reference image, depth as position embeddings, and one token per LiDAR bin. It outputs base waypoints and camera tilt 2411.14770 §III-B.

In simulation the completion rate is 95.9% when the object is visible at the start, against 89.0% for a baseline that uses a CAD model, a visible object and a perfect map 2411.14770 fig. 6 2411.14770 §V-A. The reported 3 cm and 1 degree median errors come only from runs that finish within 1 m 2411.14770 §V. In a real kitchen, with four frames of history, three of three runs completed for five of six objects. With no history, the same objects completed between one and three runs 2411.14770 tab. III. The paper says history matters on the robot and gives no benefit in simulation 2411.14770 §IV 2411.14770 §V-C.

Noise on executed commands while recording, and a geometric gate

LiCS records a planner's velocity commands, and the human joystick runs, in Gazebo. Gaussian noise with σ = 0.25 is added to the command the simulator executes, but the recorded target is the clean command 2406.14947 §II-A 2406.14947 §IV-A. The network is a Transformer that reads a 2D LiDAR scan and a normalised local goal 2406.14947 §II-B.

On a Jackal, a geometric check on the straight and turning swept regions guards the motion. When it flags a motion, the layer slows down, rotates in place or stops, using handcrafted rules 2406.14947 §II-C. The check runs only in the hardware trials 2406.14947 §IV-D. On three cardboard tracks, the robot completed two, with five trials each. The third was not completed, and the paper blames collisions and stuck corners 2406.14947 tab. 5 2406.14947 §IV-D. In simulation at 1.4 m/s, the policy completes 100% of 66 benchmark worlds, against 99.49% for the planner it copies 2406.14947 tab. 2.

Human wrist plans, distillation and augmentation for a dexterous hand

A Transformer planner, trained by behaviour cloning on human wrist motion, proposes 6-DoF wrist targets. A PPO controller adds finger commands and a wrist residual limited to ±4 cm and ±0.5 rad 2411.04005 §4.1 2411.04005 §4.2. Joint velocities and object velocities cannot be measured on the real hand, so a teacher that sees them is distilled into the policy with DAgger. Object pose comes from four top-down cameras, using FoundationPose at 15 Hz 2411.04005 §4.4 2411.04005 app. C.

In simulation, single-object policies complete 83.8% of a trained trajectory with the augmentation loop, and 59.6% without it 2411.04005 tab. 4. In six real-world tasks, the policy completes 64.3% to 100% of each trajectory, over 20 trials per rate. The vanilla RL baseline completes 4.5% to 56.1% 2411.04005 tab. 2. The paper's domain randomisation table lists 16 settings, such as object and robot mass, friction, joint stiffness and gravity 2411.04005 app. E. The released configuration turns randomisation off and uses different ranges, so the table is not the code's default the card. The public repository has the RL environment, but not the planner or the augmentation loop the card.

Start a dexterous hand from a grasp prior

An AnyGrasp two-finger grasp on the object's functional part is mapped to a 16-DoF Allegro hand and filtered for collisions. The grasp closest to the camera is the start state, and then PPO refines the arm and hand from it 2412.15587 §3.1 2412.15587 fig. 2. In SAPIEN, on an XArm6, the method beats DexArt on all four tasks, for example 94.17% against 79.17% on the bucket 2412.15587 tab. 1. On a UR5 with the same hand, the bucket and laptop tasks reach 79.08% and 74.31% averages, over about ten trials each 2412.15587 tab. 2. The real-robot policy is retrained in simulation with the real camera pose, and the setup uses foam boards and a nail as props 2412.15587 §4.2.

Simulation-only policy with privileged critic

A drone team with a camera and a cable-slung payload is trained by PPO with an asymmetric actor-critic. The critic sees the full simulated state, and the actor sees partial observations 2409.03930 §3.2. This is the same split as in the Shadow Hand task above. In Gazebo the method reaches success rates of 0.824 to 0.903, against 0.502 to 0.786 for Q-learning, DQN and Sarsa 2409.03930 tab. 1. The reward, observation, action space, network and training budget are not stated, so the policy cannot be rebuilt from the paper alone 2409.03930 §3.2. The paper says its experiments cover both simulation and the real world, but the source reports only simulation 2409.03930 §4 2409.03930 §5.

Make the simulated site harsher than the real one

A quadruped learns to cross pipes in Isaac Gym, with the walls set to low friction (static 0.2, dynamic 0.1), a raised entrance and random pushes on the base 2412.13621 §II-A4. The paper groups these under sim-to-real modifications. Friction is also randomised between 0.6 and 2, and 0 to 3 kg is added to the base mass 2412.13621 §III-A. Up to four obstacles of 0.1 to 0.3 m high are placed on the pipe surface in the last stage 2412.13621 §II-A3 2412.13621 tab. III. Training uses a height scan of the floor and the ceiling, which the simulator provides and the robot does not 2412.13621 §II-B2. The paper's second stage, a depth-camera student, is shown only in a figure, with no training details 2412.13621 fig. 1.

In simulation, the 0.2 m pipe with obstacles succeeds in 93.75% of 16 runs 2412.13621 tab. IV. On three real PVC pipes of 0.19 to 0.24 m radius, the robot crossed without obstacles in every trial. On the smallest pipe with obstacles, it crossed in 46.5% of trials 2412.13621 tab. V. The number of real trials is not stated, and the paper says real success was much lower than in simulation with the same policies 2412.13621 §III-D.

Keypoint tracker shared across humanoid tasks, with a depth-based task generator

The tracker's command is an 18-value vector: the root's position error, plus the offsets of the head, both hands and both feet from the root [arxiv:2509.20322 §3.1] 2509.20322 eq. 1. A tracker is trained once on human motion. A teacher sees the reference motion for the next 2 seconds and privileged contact forces, and a student that sees proprioception and the command is distilled from it with DAgger 2509.20322 §3.1. Each task then gets its own generator, which outputs the command. Its teacher sees object state and is trained with PPO on a task reward. Its student reads depth through a CNN and is distilled with DAgger 2509.20322 §3.2. Because the tracker is shared, a new task retrains only the generator 2509.20322 §3.

Two stabilisers are added. During tracker training, each command dimension is multiplied by a factor drawn from U(0.5, 1.5). The generator's output is clipped to a band of μ ± 1.64σ, where μ and σ are estimated from human motion 2509.20322 §3.3. The ablations in simulation say that removing the noise breaks the push-box task, and that removing the clip makes returns collapse late in training 2509.20322 §4.5 2509.20322 tab. 3. Real depth is approximated with random masks on simulated depth. The camera tilt is randomised by ±5°, and a binary command pauses the task half of the time during training 2509.20322 §3.4.

On a Unitree G1 with a RealSense D435i, the policies lift a box, push a box, dribble a ball and kick a box, indoors and outdoors. The paper gives no success rates for these runs 2509.20322 §4.2. In simulation, the vision student covers 37 m in one-minute push-box rollouts, against 152 m for the state-based teacher, and 135 m against 189 m on the ball-kick task 2509.20322 tab. 2. Without vision, the student covers 2 m on the push box 2509.20322 tab. 2. The released code is a MuJoCo sim2sim demo with four checkpoints, and the training code is not in it the card.

Survey taxonomies

Two surveys sort the methods into families, and neither reports transfer numbers. A small-car survey lists four routes: zero-shot transfer, transfer learning, domain adaptation and domain randomisation 2404.06229 §2.3. It notes that domain randomisation is offered on two platforms, and reports no transfer result for any platform the card. A legged-locomotion survey covers avoiding simulator exploits, system identification, domain randomisation and domain adaptation, the last by inferring the environment 2406.01152 §5.1 2406.01152 §5.2 2406.01152 §5.3 2406.01152 §5.4. It names mass, friction, motor strength and latency as randomised quantities, with no ranges 2406.01152 §5.3. It contains no experiments of its own the card.

When to use which

The rows use constraints from each paper's own setup. The robot results come from one robot, the humanoid results from one Digit V3 and MuJoCo, the manipulation results from one arm, the grasp results from one Franka and one depth camera, the hand results from one TriFinger and simulation, and the parking results from one vehicle, so every hardware claim below holds for those systems alone.

Implementation sketch

Each block is one step from one source. They are not a single pipeline. Values are the paper's where it gives them.

# Add uniform noise to proprioception for the actor  [arxiv:2601.07718 tab. 4]
def noisy_proprio(s):
    s.ang_vel += uniform(-0.2, 0.2)       # rad/s
    s.gravity += uniform(-0.05, 0.05)
    s.joint_pos += uniform(-0.01, 0.01)   # rad
    s.joint_vel += uniform(-0.5, 0.5)     # rad/s
    return s

# Give the actor the depth frame from one step earlier  [arxiv:2601.07718 §3.2]
def depth_for_actor(depth_buffer):
    return depth_buffer.previous()

# The critic sees the clean state and base linear velocity  [arxiv:2601.07718 tab. 3]
def critic_input(clean_state, base_lin_vel):
    return concat(clean_state, base_lin_vel)

# Sample humanoid dynamics for each episode; ranges come from the caller
# Quantities named in the paper; no values given  [arxiv:2408.07295 §4.5]
def randomise_humanoid(sim, ranges):
    sim.joint_damping = sample(ranges.damping)
    sim.link_mass = sample(ranges.mass)
    sim.com_offset = sample(ranges.com)
    sim.encoder_noise = sample(ranges.encoder)
    sim.ground_friction = sample(ranges.friction)

# Stage 1 torso push: one step, 80 to 800 N  [arxiv:2408.07295 §4.4]
# Each policy step has a 1% chance of a push
def stage1_push():
    if uniform(0, 1) < 0.01:
        return uniform(80, 800)   # N
    return 0.0

# Add Gaussian noise to simulated depth before building the point cloud
# [arxiv:2103.14127 fig. 6]; the paper does not state sigma
def noisy_depth(depth, sigma):
    return depth + normal(0.0, sigma, size=depth.shape)

# Resample simulator parameters at each reset  [arxiv:2108.10470 tab. 18]
def randomise_on_reset(sim):
    # Scaling factors drawn uniformly  [arxiv:2108.10470 tab. 18]
    sim.object_size *= uniform(0.95, 1.05)
    sim.link_mass *= uniform(0.5, 1.5)
    sim.friction *= uniform(0.7, 1.3)
    # Scaling factors drawn log-uniformly  [arxiv:2108.10470 tab. 18]
    sim.joint_damping *= loguniform(0.3, 3.0)
    sim.p_gain *= loguniform(0.75, 1.5)
    # Additive terms drawn from normal distributions  [arxiv:2108.10470 tab. 18]
    sim.joint_limit_offset = normal(0.0, 0.15)   # rad
    sim.gravity += normal(0.0, 0.4, size=3)      # m/s^2, per coordinate

# Pre-train on synthetic images, then fine-tune on real ones  [arxiv:1810.08705 §5.1]
def train_two_stage(model, synthetic_set, real_set):
    # Stage 1: synthetic labels only  [arxiv:1810.08705 tab. 4]
    train(model, synthetic_set)
    # Stage 2: continue on the real set from the synthetic weights  [arxiv:1810.08705 §5.1]
    train(model, real_set)
    return model

# After one real trial, update the visual adapter only  [arxiv:2312.16217 §3.3]
def adapt_after_trial(model, rgb, pixel, succeeded):
    # Ask the yes/no affordance question about the predicted point  [arxiv:2312.16217 §3.3]
    question = affordance_question(rgb, pixel)
    # The real outcome is the answer target  [arxiv:2312.16217 §3.3]
    target = "yes" if succeeded else "no"
    loss = model.loss(rgb, question, target)
    model.step_visual_adapter(loss)

# Student objective: RL loss plus a KL term to the teacher  [arxiv:2303.03381 eq. 2]
# The weight lam is annealed to zero near mid-training; its shape is not stated
# [arxiv:2303.03381 § Joint optimization with reinforcement learning]
def student_objective(rl_loss, kl_term, lam):
    return rl_loss + lam * kl_term   # KL direction as written in eq. 2

# Record a step with noise on the executed command  [arxiv:2406.14947 §II-A]
# The stored label is the clean expert command, not the noisy one
def record_step(lidar, goal, expert_cmd):
    # Gaussian noise on (v, omega); sigma = 0.25 in the paper, units not stated
    noise = normal(0.0, 0.25, size=2)
    executed = expert_cmd + noise          # what the simulated robot does
    sample = (lidar, goal, expert_cmd)     # what the network learns to copy
    return executed, sample

Contradictions

No two sources report conflicting results for the same setup. The disagreements below sit inside single sources, and each side is cited.

• KL direction. Equation 2 writes the student term as D_KL(π_o ‖ π_s) 2303.03381 eq. 2. The text describes distilling the teacher "through" KL divergence [arxiv:2303.03381 § Teacher state-policy supervision]. Forward and reverse KL differ, so this page follows eq. 2 as written. The card asks the implementer to check the supplement the card.

• Standing against SaW. The text calls the Digit V3 policy "on par" with the Stand-and-Walk baseline 2408.07295 §5.3. The table shows SaW drifts less under a 50 N push (0.109 m against 0.256 m) 2408.07295 tab. 2. The policy drifts less under 100 N and when walking.

• Best on setA. The text says MHC_A "excels on setA" 2408.07295 §5.4. The table gives MHC_AB the best setA score on both MEPE and root drift 2408.07295 tab. 4. The card follows the table.

• Hardware validation. The Digit perception paper's abstract describes a preliminary hardware validation the card. The body compares simulated and real depth images only, and states that no policy ran on hardware 2512.12993 fig. 12 2512.12993 §V. Read the claim as a sensor comparison.

• Synscapes fine-tuning gain. The text says Synscapes gives "more than twice the gain compared to both Richter and Synscapes" 1810.08705 §5.1. The table gives +2.29 for Synscapes, +1.01 for the GTA-based set and +0.89 for Synthia, against the 76.56 baseline 1810.08705 tab. 4. The card records that the second name in the sentence is probably a typo for Synthia. Check the table, not the sentence.

• Depth noise. The paper's simulated pipeline adds range noise, disparity artifacts and out-of-distribution replacement 2601.07718 app. A.4. The released parkour training configuration applies only crop, blur and normalisation, according to the card.

• Edge penalty constant. Equation 1 uses ε = 10⁻³ 2601.07718 eq. 1. Appendix A.5 lists 10⁻⁶ 2601.07718 app. A.5. The released code uses 10⁻⁶, according to the card.

• Simulator camera rate. Hardware depth runs at 60 Hz 2601.07718 §3.2. The released simulation updates the camera every 0.02 s, according to the card.

• Simulated parking success. The main table gives REAP-SAC 83.5% on standard slots 2605.08713 tab. 1, and the ablation baseline repeats that number 2605.08713 tab. 2. The simulator table gives the same 3DGS method 82.5% 2605.08713 tab. 3. The paper does not explain the gap, and the ablation table does not say which slot type it covers 2605.08713 tab. 2.

• Success thresholds. ManipLLM's paper gives thresholds of 0.01 and 0.1 2312.16217 §4.1, but not the cosine rules. The code's training labels and test rule differ, according to the card. Compare results only under one rule.

• Repair gain. The MARTENS abstract gives 25% to 50% more failures than random search, and its conclusion gives 26% to 50%. The body gives 50% for UC-1 and 26% for UC-2 2410.19277 §1 2410.19277 §3.2.1. The printed failure counts give 49% and 26% 2410.19277 tab. 2 2410.19277 tab. 3. This page quotes the body's figures.

• Fine-tuning epochs. The MARTENS abstract says the fine-tuned models reach their scores in fewer than 10 epochs. Table 8 gives 12 epochs for the UC-2 model that starts from the repaired detector 2410.19277 tab. 8.

• Dexterous hand, text and table. The text says the method improves on vanilla RL by more than 50% on the real objects. Table 2 gives a 39.4-point gap on the microwave against fingertip mapping 2411.04005 §5.4 2411.04005 tab. 2. The page quotes the table.

• Grasp-prior headline numbers. The text gives 92%, 87% and 87% for the bucket, laptop and toilet at later training steps. Table 1 gives 94.17, 90 and 87.88 for the same rows 2412.15587 §4.1.3 2412.15587 tab. 1. The paper does not say which checkpoint each figure comes from.

• Push-box mass and speed. The paper's vision policy pushes a 3.8 kg box on hardware. The 37 m per minute figure matches the simulated 4 kg Push Box row, and the paper does not say which box it measures 2509.20322 tab. 2. The card records the same mismatch the card.

• Real-world claim without results. The drone paper says its method was shown in both simulation and the real world. The source reports only simulation 2409.03930 §4 2409.03930 §5.

• Hardware speed. The LiCS abstract gives 1.5 m/s as the safe maximum. Its hardware section says 1.5 m/s was tried and 1.3 m/s performed best, without saying which trials used which speed 2406.14947 §IV-D. The card records the abstract's wording the card.

Papers

• High precision grasp pose detection in dense clutter: a 15-channel CNN grasp classifier, pretrained on CAD-rendered depth and tested on a Baxter with author-reported 93% grasp success.
• Sim-to-Real Learning for Bipedal Locomotion Under Unsensed Dynamic Loads: a proprioception-only PPO walker for the Cassie biped, trained with four dynamic loads in simulation and tested on hardware at lower speeds.
• Learning Sim-to-Real Humanoid Locomotion in 15 Minutes: an off-policy recipe (FastSAC and FastTD3) for massively parallel humanoid training, with a G1 whole-body tracking hardware test.
• Learning Terrain Aware Bipedal Locomotion via Reduced Dimensional Perceptual Representations: a height-map VAE latent for a Digit walker, distilled into a depth-camera encoder, with simulation results only.
• Hiking in the Wild: A Scalable Perceptive Parkour Framework for Humanoids: a humanoid parkour policy trained in simulation with depth and an edge-contact penalty, reported zero-shot on a Unitree G1.
• Grasp Pose Detection in Point Clouds: a sampler and 15-channel CNN for grasps on depth clouds, pretrained on simulated exemplars and fine-tuned on real BigBird scans, with a Baxter clutter test.
• Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes: a per-point grasp network trained on simulated grasps, with depth noise and a real Franka test using an Intel RealSense L515.
• Real-World Humanoid Locomotion with Reinforcement Learning: a causal-transformer walker for the full-size Digit, trained in simulation with PPO and a KL term to a state-based teacher, deployed zero-shot outdoors.
• Learning Multi-Modal Whole-Body Control for Real-World Humanoid Robots: one masked-directive LSTM policy for a Digit V3 humanoid, trained in MuJoCo with domain randomization and a three-stage curriculum, with hardware trials and no released code.
• Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing: a 25,000-image synthetic street dataset, with results for pre-training and fine-tuning on Cityscapes and KITTI.
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning: a GPU physics simulator for robot learning, with a domain-randomised Shadow Hand task and sim-to-real claims on ANYmal and a TriFinger.
• ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation: a multimodal model fine-tuned on simulated articulated objects, tested on a real arm with test-time adaptation.
• REAP: Reinforcement-Learning End-to-End Autonomous Parking with Gaussian Splatting Simulator for Real2Sim2Real Transfer: a camera-to-action parking policy trained with SAC in a scanned Gaussian-splatting simulator, reported on a real vehicle, with no released code.
• Harmonic Mobile Manipulation: one PPO policy for base and arm on a Stretch robot, trained in ProcTHOR and run zero-shot in one real apartment, with Apache-2.0 code that differs from the paper in several values.
• Autonomous Driving Small-Scale Cars: A Survey of Recent Development: a survey of small research car platforms and their tasks, with no experiments and no transfer numbers.
• Learning-based legged locomotion; state of the art and future perspectives: a survey of learned legged locomotion, covering privileged teacher-student training and the sim-to-real methods, with no experiments.
• LiCS: Navigation using Learned-imitation on Cluttered Space: a small Transformer copies a local planner from simulated data, with noise on executed commands and a geometric check on a Jackal.
• VisualMimic: Visual Humanoid Loco-Manipulation via Motion Tracking and Generation: a shared keypoint tracker from human motion and a depth-based task generator, with qualitative G1 hardware runs and a MuJoCo sim2sim demo release.
• DRAL: a PPO drone team with an asymmetric actor-critic, evaluated in Gazebo only.
• In-Simulation Testing of Deep Learning Vision Models in Autonomous Robotic Manipulators: a genetic search over an Isaac Sim work cell finds detector failures, which are then repaired and tested on a few real images.
• Object-Centric Dexterous Manipulation from Human Motion Data: a human-data wrist planner and a PPO finger controller for two dexterous hands, with distillation, domain randomisation and six real objects.
• Aim My Robot: Precision Local Navigation to Any Object: a transformer that reaches a centimetre-precise pose beside an object, trained in simulation with DAgger and tested in a real kitchen.
• Learning Quadrupedal Robot Locomotion for Narrow Pipe Inspection: a Go2 quadruped trained in Isaac Gym with a privileged height scan and harsher simulated pipes, tested on three real PVC pipes.
• Dexterous Manipulation Based on Prior Dexterous Grasp Pose Knowledge: a grasp-prior start followed by PPO refinement for an Allegro hand, with simulation results and a small real UR5 test.