Pretraining Synthetic Data Generation

Creator

Creator

Seonglae Cho

Created

Created

2025 Mar 6 16:44

Editor

Editor

Seonglae Cho

Edited

Edited

2025 Nov 7 11:37

Refs

Refs

Only Portion of pretraining

How to avoid collapese: ToEdit (token level edit)

https://arxiv.org/pdf/2412.14689

Cosmopedia for

SmolAgents LM useful to

On-device AI small llm

Cosmopedia: how to create large-scale synthetic data for pre-training Large Language Models

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Cosmopedia: how to create large-scale synthetic data for pre-training Large Language Models

https://huggingface.co/blog/cosmopedia

Cosmopedia: how to create large-scale synthetic data for pre-training Large Language Models

Unconditional Generation

FaithfulSAE: Towards Capturing Faithful Features with Sparse...

Sparse Autoencoders (SAEs) have emerged as a promising solution for decomposing large language model representations into interpretable features. However, Paulo and Belrose (2025) have highlighted...

https://www.arxiv.org/abs/2506.17673

https://arxiv.org/pdf/2510.18554

Recommendations

/////////