Seonglae ChoPPL
Important metric for language modeling is validation perplexity, which is a representative of upstream quality. However, since it does not guarantee the performance of the downstream task, it should be checked separately. In other words, a low PPL value means high probability on data, but it does not necessarily mean a good language model.
Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized sequence .
Calculating Perplexity for predetermined input text is more common. When measuring the context length limit to determine how much tolerance a model has for context length, it is assessed during generation.
Property
- When perplexity is high, the model tends to have flat attention scores rather than focusing on specific tokens, while with low perplexity, it shows sharp attention patterns focused on relevant tokens
Perplexity Notion
Perplexity of fixed-length models
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
https://huggingface.co/docs/transformers/perplexity
03-05 펄플렉서티(Perplexity, PPL)
두 개의 모델 A, B가 있을 때 이 모델의 성능은 어떻게 비교할 수 있을까요? 두 개의 모델을 오타 교정, 기계 번역 등의 평가에 투입해볼 수 있겠습니다. 그리고 두 모델이 해…
https://wikidocs.net/21697
Bullshit Receptivity Scale, CBRS(Corporate Bullshit Receptivity Scale)
Lack of Critical Thinking (Reflective Thinking) or low Intellect with too much Openness makes one vulnerable to bullshit and fake news. Be careful about profound phrases such as for self-development Aphorism with lots of abstract concepts. Prefer Straightforward expression.
On the reception and detection of pseudo-profound bullshit | Judgment and Decision Making | Cambridge Core
On the reception and detection of pseudo-profound bullshit - Volume 10 Issue 6
https://www.cambridge.org/core/journals/judgment-and-decision-making/article/on-the-reception-and-detection-of-pseudoprofound-bullshit/0D3C87BCC238BCA38BC55E395BDC9999
BSR Test
gordonpennycook.com
https://gordonpennycook.com/wp-content/uploads/2023/09/the-bullshit-receptivity-scale.pdf