Title: Priors in Bayesian neural networks at the unit level
Authors: Mariia Vladimirova - Inria (France) [presenting]
Julyan Arbel - Inria (France)
Abstract: Neural networks (NNs), and their deep counterparts, have largely been used in many research areas such as image analysis, signal processing, or reinforcement learning, to name a few. The impressive performance provided by such machine learning approaches has greatly motivated research that aims at a better understanding of the driving mechanisms behind their effectiveness. In particular, the study of the NNs distributional properties through Bayesian analysis has recently gained much attention. We firstly describe the necessary notations and statistical background for Bayesian NNs. Then we consider its distributional properties and novel theoretical insight on distributions at the units level. Under the assumption of independent and normally distributed weights, we establish that the induced prior distribution on the units before and after activation becomes increasingly heavy-tailed with the depth of the layer.