Seonglae ChoGeneralization bounds are a safety check
They give a theoretical guarantee on the performance of a learning algorithm on any unseen data.
Just because a model fits well on training data doesn't guarantee it will perform well in practice. However, we can mathematically bound how well it generalizes.
The PAC-Bayes Bound provides a probabilistic guarantee for this, showing that the difference between training error and test error can be expressed as a complexity term based on the Divergence Distance between the Prior and Posterior. This can be simply expressed as follows:
where is sample count and is the probability of being misled by the training set.
- Bound depends on the distance between prior and posterior
- Better prior (closer to posterior) would lead to tighter bound
- Learn the prior P with part of the data
- Introduce the learnt prior in the bound
Dziugaite and Roy (2017), Neyshabur et al. (2017) have derived some
of the tightest deep learning bounds in this way
2017
2024