The Simulation Illusion
We trained a robotic arm using Proximal Policy Optimization (PPO) in a MuJoCo physics simulation to sort recycling. In the simulation, it achieved a 99% success rate. The loss curves were perfect.
We deployed the weights to the physical arm in the lab. On its first attempt, it aggressively smashed a glass bottle into the side of the bin, destroying the end-effector.
Bridging the Gap
The physics engine didn't perfectly model real-world friction, gear backlash, or actuator delay. Furthermore, the camera feed had different noise profiles than the cleanly rendered simulation.
We had to implement Domain Randomization.
# Concept: Randomize simulator parameters every episode
def reset_environment(env):
# Randomize lighting for camera robustness
env.set_lighting(random.uniform(0.5, 1.5))
# Randomize physical parameters to prevent overfitting to physics engine
env.set_friction_coefficient(random.uniform(0.8, 1.2))
env.set_mass("target_object", random.uniform(0.5, 2.0))
return env.reset()By drastically randomizing lighting, friction, and mass in the simulation during training, we forced the RL agent to learn robust, generalized recovery behaviors rather than memorizing the exact physics of the simulator.