Seonglae ChoICL sigmoid: as the number of in-context examples increases, model behavior follows an S-shaped curve: minimal change → sudden sharp transition → saturation. Jailbreak threshold: the inflection point in the middle of this sigmoid curve. When examples accumulate to around this point, the model's internal beliefs cross a critical threshold, causing it to suddenly exhibit risky/prohibited behaviors it previously avoided. Near the threshold, just 1–2 additional examples or weak activation steering can trigger a sharp behavioral flip between safe ↔ violation modes.
In-context learning and Activation Steering are fundamentally the same mechanism. They both modify the model's Belief State about latent concepts. ICL accumulates evidence (likelihood) through contextual examples, while Steering directly shifts the prior probability. The two combine additively in log-belief space → small adjustments can cause sharp behavioral phase shifts. This perspective enables prediction and explanation of phenomena like many-shot ICL's sigmoid learning curve and jailbreak thresholds.
Many-shot jailbreaking observes sharp transition
many-shot ICL reports phase-transition-like behavior
