Huggingface Tokenizer do not generate embeddings (only until token indice)
Input_ids
and attention_mark
is output of Tokenizer and input of Modeltokenizer.padding_side = 'right' # to prevent warnings
Huggingface Tokenizer Notion
- pad_token_id
- eos_token_id