The Reinforcement Learning Mirage
Reinforcement Learning in quant trading is 90% reward shaping and 10% model architecture. We’ve been running a Proximal Policy Optimization (PPO) agent for order execution, attempting to beat standard TWAP/VWAP baselines on volatile tech equities.
The biggest issue? The agent is incredibly good at exploiting the simulator. In our first run, it found a floating-point rounding bug in our slippage model and essentially "printed money" in backtests by bouncing bid-ask spreads infinitely.
Rebuilding the Simulator
Once we fixed the simulator using high-fidelity Level 3 order book data (specifically leveraging the Nasdaq ITCH Data from Kaggle), the agent stopped learning entirely. The noise-to-signal ratio in real L3 data was too high for the value network to converge.
# GAE Configuration snippet in RLlib
config = (
PPOConfig()
.training(
gamma=0.99,
lr=3e-5,
clip_param=0.1,
# Crucial for noisy financial data:
use_gae=True,
lambda_=0.90, # Low lambda to prioritize short-term actual returns
vf_clip_param=10.0,
)
.environment("OrderExecutionEnv-v0")
)Advantage Estimation Tuning
We had to heavily clip the rewards and use Generalized Advantage Estimation (GAE) with a very low lambda ($0.90$) to stabilize the value network. In high-frequency environments, a high lambda relies too much on long-term bootstrapped values, which are essentially random walks. By lowering lambda, we forced the agent to focus on immediate execution costs.