EcoSta 2019: Start Registration
View Submission - EcoSta2019
Title: Model selection bias invalidates goodness of fit tests Authors:  Joshua Loftus - New York University (United States) [presenting]
Weichi Yao - New York University (United States)
Abstract: Goodness of fit tests are studied in a variety of model selection settings and find that selection bias generally makes such tests conservative. Since selection methods choose the ``best'' model, a goodness of fit test will usually fail to reject, even when the incorrect model has been chosen. This is troubling, as it implies these tests in practice do not actually provide evidence in favor of the chosen model. We also explore methods of post selection inference to obtain conditionally unbiased goodness of fit tests and show how these outperform the unadjusted tests.