Byte pair encoding
Method of finding meaningful prefixes or suffixes by separating at the character level
It can alleviate the Out-Of-Vocabulary problem
Set a predefined dictionary size or number of merges (K), or continue merging until the maximum pair frequency falls below a certain threshold.
BPE Notion
