Seonglae ChoDocument texts are a sample from the language model
Missing words should not have zero probability of occurring. Smoothing is a technique for estimating probabilities for missing (or unseen) words
- Laplace smoothing (Laplace correction)
Discounting Methods
Laplace smoothing gives too much weight to unseen terms
- Lidstone correction
- Absolute discounting
Interpolation Methods
discounting treats unseen words equally
- Jelinek-Mercer Smoothing
- Smoothing with Background probabilities
- Dirichlet Smoothing