Initially, it relies on simple memorization based on learned concepts, but at a certain point it suddenly develops the ability to generate new concept combinations even in Out-of-Distribution (OOD) situations
Neural Network Phase Change
toy model experiment basically
