The Deep Learning Fallacy
I spent the weekend analyzing the top 10 solutions from recent Kaggle tabular data competitions. Every few months, a new paper comes out claiming that Transformer-based tabular architectures (like TabNet or FT-Transformer) have finally beaten tree-based models.
Despite the academic hype, 9 out of 10 winning solutions in the real world were massive ensembles of LightGBM, XGBoost, and CatBoost.
Why Trees Win
Gradient Boosted Trees (GBDTs) handle unscaled data, categorical variables, and missing values gracefully out of the box. Neural networks require meticulous normalization, feature embedding, and are incredibly prone to overfitting on the sharp decision boundaries typical of tabular datasets.
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold
# The gold standard for tabular data in 2026
params = {
'objective': 'binary:logistic',
'tree_method': 'hist', # GPU accelerated histogram binning
'learning_rate': 0.01,
'max_depth': 6,
'colsample_bytree': 0.8,
'subsample': 0.8
}
dtrain = xgb.DMatrix(X_train, label=y_train)
model = xgb.train(params, dtrain, num_boost_round=5000, evals=[(dvalid, 'eval')], early_stopping_rounds=100)The Takeaway
If you have structured, heterogeneous data, don't waste your GPU hours tuning deep neural networks. Spend that time engineering better features. The only exception is if your tabular data contains high-cardinality categorical features that can benefit from learned embeddings, in which case a wide-and-deep hybrid approach is warranted.