Seonglae ChoPersona Features control Emergent Misalignment
Even with small datasets, when fine-tuning LLMs through SFT or reward-based reinforcement learning (RL), unintended "broad" malicious responses (emergent misalignment) can occur. Additionally, rapid re-alignment is possible with a small amount of "normal data" fine-tuning. Misaligned persona features were discovered using SAE persona features.
Misaligned persona features activate and cause misalignment, and by comparing activations between original and misaligned models, related features can be identified and directly manipulated (steering/ablation) to verify their causal role. Ablating just 10 misalignment-related features significantly reduces the rate of misaligned responses while maintaining coherence.
