Initially, it relies on simple memorization based on learned concepts, but at a certain point it suddenly develops the ability to generate new concept combinations even in Out-of-Distribution (OOD) situations
Phase Change

toy model experiment basically

arxiv.org
https://arxiv.org/pdf/2406.19370

Seonglae Cho