Shannon Game

Creator

Created

2025 Mar 19 11:37

Editor

Edited

2025 Mar 19 11:43

Refs

The information content of a message is a function of how predictable it is. The information content (number of bits) needed to encode i is . So

Next Token Prediction probability is containing information content itself.

The entropy of a message is the expected number of bits needed to encode it. (

///////