Tiktoken

Creator

Creator

Seonglae Cho

Created

Created

2023 Jun 11 4:34

Editor

Editor

Seonglae Cho

Edited

Edited

2026 Jul 26 22:47

Refs

Refs

Tiktoken Usages

Tiktoken Encoding


encoder = encoding_for_model('gpt-4')
tokens = encoder.encode(text)
print(tokens.length)

https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

tiktoken (OpenAI) tokenizer — 🦜🔗 LangChain 0.0.194

https://python.langchain.com/en/latest/modules/indexes/text_splitters/examples/tiktoken.html

Recommendations

///////