num_proc- good number to use is ~order number of cpu cores // 2- but it needs dataset contains multiple shards
Huggingface Datasets load_dataset()
Creator
Creator
Seonglae ChoCreated
Created
2023 Nov 29 6:52Editor
Editor
Seonglae ChoEdited
Edited
2024 Feb 29 9:20Refs
Refs