A1248
Title: The revival of bagged trees: Hierarchical shrinkage as a regularization tool
Authors: Markus Loecher - Berlin School of Economics and Law (Germany)
Arne Gevaert - Ghent University (Belgium)
Bastian Pfeifer - Medical University of Graz (Austria)
Sourav Adhikari - Berlin School of Economics and Law (Germany) [presenting]
Andreas Holzinger - BOKU (Austria)
Abstract: Random forests are among the most widely used methods for tabular prediction, yet the interactions between implicit regularization and explicit post-hoc regularization are not fully understood. Bagged trees and random forests are revisited through the lens of hierarchical shrinkage, a post-hoc method that shrinks node predictions towards ancestor means. Hierarchical shrinkage is a useful tool for regularizing tree-based models and helps clarify the relationship between bagging and random forests, observed across different levels of signal-to-noise ratios. Building on degrees of freedom arguments for random forests, shrinkage, tree depth, and feature subsampling are examined, along with their effects on model flexibility and prediction. An adaptive variant of hierarchical shrinkage is introduced that adjusts for the amount of shrinkage according to split informativeness and feature entropy, addressing insensitivity to overfitting and bias towards high cardinality variables. Through simulations, shrinkage is shown to improve bagged trees, partially recovering the benefits of expensive hyperparameter tuning, and offering performance comparable to tuned random forests. In addition, the proposed adaptive shrinkage improves the stability and credibility of feature importance assessments derived from SHAP values and mean decrease in impurity, placing hierarchical shrinkage as both a practical regularization tool and a bridge between predictive performance and interpretability.
