Seonglae ChoAI Feature Dimensionality
Is there a way we could understand what "fraction of a dimension" a specific feature gets?
Perhaps the most striking phenomenon the Anthropic have noticed is that the learning dynamics of toy models with large numbers of features appear to be dominated by "energy level jumps" where features jump between different feature dimensionalities.
