Title: Bayesian loss function selection with the Hyvarinen score
Authors: Jack Jewson - Universitat Pompeu Fabra and Barcelona Graduate School of Economics (Spain) [presenting]
David Rossell - Universitat Pompeu Fabra (Spain)
Piotr Zwiernik - Universitat Pompeu Fabra (Spain)
Abstract: General Bayesian updating provides a coherent procedure for updating beliefs about the minimiser of a loss function which need no longer index a probability density. Importantly this allows for Bayesian learning using algorithms as well as models. However, methods for selecting which amongst a series of loss functions is more appropriate for the data at hand are currently primitive. Loss functions need no longer define normalised probability densities and are thus no longer scale-invariant. As a result, standard Bayesian model selection tools do not apply. Instead, we appeal to the homogeneity property of the Hyvarinen score to select between loss functions in a scale-invariant manner. The chosen loss function can be interpreted as the pseudo-probability model that best captures the data generating process's relative probabilities. Doing so guarantees consistency, meaning we are still able to detect the data generating model if it is under consideration. In particular, we focus on examples from robust regression, where we are able to estimate the hyperparameter of Tukey's loss, and binary classification, comparing the SVM to logistic regression.