Seonglae ChoTransformer Interpretability
A transformer starts with a token embedding, followed by a series of residual blocks, and finally a token unembedding
MLPs store factual information and Attention performs pattern matching. Since in transformer models, MLPs are token-wise operations while attention is an inter-token operation
Both the attention and MLP layers each “read” their input from the residual stream (by performing a linear projection), and then “write” their result to the residual stream by adding a linear projection back in.
- Transformers have an enormous amount of linear structure.
- One layer attention-only transformers are an ensemble of bigram and “skip-trigram
- Two layer attention-only transformers can implement much more complex algorithms using compositions of attention heads
When there are many equivalent ways to represent the same computation, it is likely that the most human-interpretable representation and the most computationally efficient representation will be different. Composition of attention heads is the key difference between one-layer and two-layer attention-only transformers
- logit
- token vectot
- embedding matrix
- MLP output
- Attention head
- token unembedding
Fundamentally, if activations in attention appears linearly, then the activation function's non-linearity is a side effect, and like in diffusion models, noise reduction through layers might be the main role.
Reversing Transformer Notion
Attention-only transformers, which don't have MLP layers
Because they had much less success in understanding MLP layers so far (2021)
A Mathematical Framework for Transformer Circuits
Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3 , LaMDA , Codex , Meena , Gopher , and similar models. However, as these models scale, their open-endedness and high capacity creates an increasing scope for unexpected and sometimes harmful behaviors. Even years after a large model is trained, both creators and users routinely discover model capabilities – including problematic behaviors – they were previously unaware of.
https://transformer-circuits.pub/2021/framework/index.html
Induction heads - illustrated — LessWrong
Many thanks to everyone who provided helpful feedback, particularly Aryan Bhatt and Lawrence Chan! …
https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-heads-illustrated#Q_composition
interpreting GPT: the logit lens — LessWrong
This post relates an observation I've made in my work with GPT-2, which I have not seen made elsewhere. …
https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
arxiv.org
https://arxiv.org/pdf/2405.00208
Handcrafting transformer
Neel Nanda on Twitter / X
My weak guess is that it'll also be doing rotation-y stuff in some base - regular addition is just modular addition if you correct for the rounding! And I think models find it hard to represent things precisely as a direction with highly varying norm.— Neel Nanda (@NeelNanda5) June 20, 2023
https://x.com/NeelNanda5/status/1671094151633305602
LLM Training Data is Nonergodic → ICL Bayesian belief updating Belief State
In-context learning is described as the process of inferring which data-generating source is currently active. When predicting data mixed from multiple sources, an internal geometry emerges in which the strength of belief in each source shifts with contextual evidence. This was confirmed in Transformers trained on synthetic data, providing grounds for understanding features as multi-dimensional subspaces rather than single directions.
The geometry of nonergodic composition
An introduction to inference under nonergodic compositions.
https://simplex.pub/nonergodic-geometry/
Transformer Layer Model Layer Scaling
The next layer in a Transformer immediately cancels part of the change made by the previous layer. This selective-correction phenomenon was observed in 5 of 7 model families. A feature found at one layer therefore has to be distinguished as either information that persists into the final prediction or a temporary computation that will soon be erased.
LLM Layers Immediately Correct Each Other
Recent methods in language model interpretability employ techniques such as sparse autoencoders to decompose residual stream contributions into linear, semantically meaningful features. Such...
https://arxiv.org/abs/2609.07876
Progressively increasing a model's computational depth during training improves scaling efficiency. A grown 7.4B model reached a CORE score comparable to GPT-3 13B with roughly 20x less training compute.
How Model Growth, Recursion, and Boundary Operators Influence...
Scaling laws predict how loss decreases with increases in computation. We show, contrary to conventional wisdom, that architectural interventions can modify scaling exponents in pre-training,...
https://arxiv.org/abs/2609.19107