Title: Weighted performance evaluation of classifiers with a noisy ground truth
Authors: Ilias Benjelloun - Université de Lorraine (France) [presenting]
Abstract: In classification, one important task is to evaluate accurately what has been learned. The goal is to obtain an estimate of how well the classifier produced by the learning algorithm performs on unseen data, or if it is better than another already established learning algorithm. The less the bias and the lower the variance, the better the estimate. Over the past few decades, many performance measures have been proposed, and different evaluation procedures and statistical tests have been developed and used. For example, with limited test data, to obtain a good estimate (with low variance) of the performance of a learning algorithm, the standard way is to perform a 10-fold cross-validation procedure. However, it is rare when the quality of the test data is taken into account. Indeed, mislabelling errors affect the estimates in terms of bias, leading to overestimate the performance of some classifiers while underestimating those of others. We are interested in taking advantage of existing denoising ensembling methods to design an evaluation procedure that is less affected by noisy data. The procedure is then used to evaluate the performance of classifiers in a setting where noise is artificially introduced in the data, and is compared with traditional evaluation procedures. The effect of different types of noise on the evaluation methods are also studied.