Seonglae ChoICL sigmoid: as the number of in-context examples increases, model behavior follows an S-shaped curve: minimal change → sudden sharp transition → saturation. Jailbreak threshold: the inflection point in the middle of this sigmoid curve. When examples accumulate to around this point, the model's internal beliefs cross a critical threshold, causing it to suddenly exhibit risky/prohibited behaviors it previously avoided. Near the threshold, just 1–2 additional examples or weak activation steering can trigger a sharp behavioral flip between safe ↔ violation modes.
Many-shot jailbreaking observes sharp transition
Improved Few-Shot Jailbreaking Can Circumvent Aligned Language...
Recently, Anil et al. (2024) show that many-shot (up to hundreds of) demonstrations can jailbreak state-of-the-art LLMs by exploiting their long-context capability. Nevertheless, is it possible to...
https://arxiv.org/abs/2406.01288
many-shot ICL reports phase-transition-like behavior
Many-Shot In-Context Learning
Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context...
https://arxiv.org/abs/2404.11018