Tiktoken Encoder.max_token_value

Creator

Created

2023 Aug 19 8:48

Editor

Edited

2023 Aug 19 8:49

Refs

dtype


dtype = np.uint16 # (can do since enc.max_token_value == 50256 is < 2**16)

/////////