Seonglae ChoTiktoken Usages
encoder = encoding_for_model('gpt-4') tokens = encoder.encode(text) print(tokens.length)
github.com
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
tiktoken (OpenAI) tokenizer — 🦜🔗 LangChain 0.0.194
https://python.langchain.com/en/latest/modules/indexes/text_splitters/examples/tiktoken.html