A0619
Title: Generalization error bound of deep learning via spectral analysis and its application to model compression
Authors: Taiji Suzuki - University of Tokyo / RIKEN-AIP (Japan) [presenting]
Abstract: To efficiently execute a high performance deep learning system on edge-computing devices, model compression methods have been gathering much attention. However, there have been a limited number of studies that simultaneously offer a practically effective compression method and its rigorous theoretical back-ground that guarantees its compression ability in connection with generalization ability. To resolve this issue, we develop a new theoretical frame-work for model compression,and propose a new pruning method called Spectral-Pruning based on the theory. We define ``degree of freedom'' to quantify an intrinsic dimensionality of the model by using the eigenvalue distribution of the covariance matrix across the internal nodesand show that the compression ability is essentially controlled by this quantity. For this bound, the theory of the kernel quadrature rule plays the essential role. Along with this, we give a sharp generalization error bound of the compressed model, and characterize a bias-variance trade-off induced by the compression procedure.We apply our method to several datasets to justify our theoretical analyses and show that the proposed method achieves the state-of-the-art performance.
