Seonglae ChoGeneralization bounds are a safety check
They give a theoretical guarantee on the performance of a learning algorithm on any unseen data.
Just because a model fits well on training data doesn't guarantee it will perform well in practice. However, we can mathematically bound how well it generalizes.
The PAC-Bayes Bound provides a probabilistic guarantee for this, showing that the difference between training error and test error can be expressed as a complexity term based on the Divergence Distance between the Prior and Posterior. This can be simply expressed as follows:
where is sample count and is the probability of being misled by the training set.
- Bound depends on the distance between prior and posterior
- Better prior (closer to posterior) would lead to tighter bound
- Learn the prior P with part of the data
- Introduce the learnt prior in the bound
Dziugaite and Roy (2017), Neyshabur et al. (2017) have derived some
of the tightest deep learning bounds in this way
2017
Exploring Generalization in Deep Learning
With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how...
https://arxiv.org/abs/1706.08947
Computing Nonvacuous Generalization Bounds for Deep (Stochastic)...
One of the defining properties of deep learning is that models are chosen to have many more parameters than available training data. In light of this capacity for overfitting, it is remarkable...
https://arxiv.org/abs/1703.11008
2024
arxiv.org
https://arxiv.org/pdf/2309.04381
openreview.net
https://openreview.net/pdf?id=GWSIo2MzuH