The Two-Step Disconnect
Typically, quantitative pipelines are disjointed. You use machine learning to predict expected returns (minimizing Mean Squared Error), and then you pass those predictions into a separate convex optimizer (like cvxpy) to allocate portfolio weights according to Markowitz Portfolio Theory.
The problem is that a model might optimize its MSE by predicting low-volatility stocks perfectly, but completely miss the high-alpha assets that the portfolio optimizer actually cares about.
End-to-End Differentiable Optimization
We built an end-to-end differentiable pipeline. Using the cvxpylayers library, we implemented the quadratic programming problem of portfolio optimization as a custom PyTorch layer inside the neural network.
import torch
from cvxpylayers.torch import CvxpyLayer
import cvxpy as cp
# Define the convex optimization problem (e.g., maximize return - risk)
# ...
portfolio_opt_layer = CvxpyLayer(problem, parameters=[expected_returns, covariance_matrix], variables=[weights])
# Forward pass
predicted_returns = model(market_features)
optimal_weights = portfolio_opt_layer(predicted_returns, current_cov_matrix)
# Loss is calculated on the actual portfolio performance, not just return prediction!
loss = -calculate_sharpe_ratio(optimal_weights, future_actual_returns)
loss.backward()This allows the gradients to flow from the final portfolio Sharpe ratio all the way back to the feature extraction layers. The model learns to predict features that specifically result in better portfolio allocations.