Huggingface Datasets

Parquet

from datasets import load_dataset, Dataset

datasets.config.IN_MEMORY_MAX_SIZE = 

Dataset.from_dict()
dataset.train_test_split(test_size=0.0005, seed=2357, shuffle=True)
dataset.select(range(100))

Create a dataset

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/docs/datasets/create_dataset

Community Eval

Community Evals: Because we're done trusting black-box leaderboards over the community

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

https://huggingface.co/blog/community-evals