Seonglae ChoExtracting alignment data (2025) - Synthetic Data Generation Template attack
Models can reproduce training data used during alignment phases (SFT, RL) either verbatim or in similar form. Since chat templates (
<|user|>, <|assistant|>) are introduced only during alignment, using them as prompts enables regeneration of alignment data through unconditional batch generation without context and only BOS or special token template prefix. Collecting model-generated data and reusing it for SFT/RL can restore performance similar to models trained on original data. Even in RL, regurgitation of training samples occurs during PPO/RLVR phases. Knowledge Distillation effectively operates as Dataset Distillation.Semantic similarity (embedding similarity ≥0.95) is defined as "semantic memorization". Traditional string similarity-based detection (Levenshtein, etc.) underestimates actual memorization rates by at least 10x.
arxiv.org
https://arxiv.org/pdf/2510.18554