Loading views...

LLM alphabetical encoding Tokenizer

Creator
Creator
Seonglae ChoSeonglae Cho
Created
Created
2025 Jan 22 12:10
Editor
Edited
Edited
2025 Jan 22 12:10
Specific
Specific
Specific
Computable
Computable
Computable
chunking 때문에 숙어 token 화 하는게 성능에는 도움될거같은데 다만 내부 alphabetic reprewsentation 을 유지하는 initialization 하던가 positional encoding 처럼 allphabetic encodign 필요하다 numberic encoding
 
 
 
 
 
 
 
 

Recommendations