Seonglae ChoSuperword BPE
SuperBPE: Space Travel for Language Models
The assumption across nearly all language model (LM) tokenization schemes is that tokens should be subwords, i.e., contained within word boundaries. While providing a seemingly reasonable...
https://arxiv.org/abs/2503.13423
