Seonglae ChoPPL
Important metric for language modeling is validation perplexity, which is a representative of upstream quality. However, since it does not guarantee the performance of the downstream task, it should be checked separately. In other words, a low PPL value means high probability on data, but it does not necessarily mean a good language model.
Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized sequence .
미리 정해진 입력 텍스트에 대한 Perplexity 계산은 더 일반적이다 생성하면서 계산하는 것에 비해. 모델이 어느정도 context length에 대해서 tolerance를 가지고 있는지 context length limit를 측정할 때에는 generation하면서 판단한다.
Perplexity Notion
