Huggingface Tokenizer

Created

Created

2023 Mar 26 1:44

Editor

Editor

Creator

Creator

Edited

Edited

2024 Mar 8 8:9

Refs

Refs

Huggingface Transformers

Huggingface Tokenizer do not generate embeddings (only until token indice)

Input_ids and attention_mark is output of Tokenizer and input of Model


tokenizer.padding_side = 'right' # to prevent warnings

Huggingface Tokenizer Notion

Huggingface Tokenizer input_ids

Huggingface Tokenizer attention_mask

Huggingface Tokenizer.apply_chat_template()

pad_token_id

eos_token_id

Recommendations

///////