Seonglae ChoCatastrophic interference
Reverse of Grokking. A tendency to abruptly lose previously learned information when learning something new. Because deep neural networks are typically trained with a cross-entropy-based loss, training focuses on fitting the distribution of the current data without explicitly accounting for the distribution of previously seen data; this is partly mitigated by scaling (more capacity).
A balance is needed between acquiring new capabilities and retaining existing ones. Grokking and catastrophic forgetting may be opposite failure modes of this trade-off, potentially captured by a single mechanistic metric: circuit stability.
Here, and are component activations for inputs and , and is the loss gradient w.r.t. that activation. Circuit stability () is then computed as the Spearman rank correlation between the resulting circuit vectors.
Even when SFT and RL achieve the same performance on a new task, RL tends to preserve prior knowledge much better. The authors argue that the extent of forgetting is driven not by the training algorithm itself, but by the amount of distributional shift.
A key predictor of forgetting is the forward KL divergence between the fine-tuned model and the base model, measured on the new-task distribution: . The authors show theoretically that, under a binary reward, policy gradients are equivalent to EM with information projection, and that on-policy RL converges to the KL-minimal optimal policy among representable policies: . This is the core idea behind RL’s Razor: among multiple solutions that solve the new task, RL selects the one closest to the base model in KL divergence.
