Seonglae ChoTransformer Interpretability
A transformer starts with a token embedding, followed by a series of residual blocks, and finally a token unembedding
MLPs store factual information and Attention performs pattern matching. Since in transformer models, MLPs are token-wise operations while attention is an inter-token operation
Both the attention and MLP layers each “read” their input from the residual stream (by performing a linear projection), and then “write” their result to the residual stream by adding a linear projection back in.
- Transformers have an enormous amount of linear structure.
- One layer attention-only transformers are an ensemble of bigram and “skip-trigram
- Two layer attention-only transformers can implement much more complex algorithms using compositions of attention heads
When there are many equivalent ways to represent the same computation, it is likely that the most human-interpretable representation and the most computationally efficient representation will be different. Composition of attention heads is the key difference between one-layer and two-layer attention-only transformers
- logit
- token vectot
- embedding matrix
- MLP output
- Attention head
- token unembedding
Fundamentally, if activations in attention appears linearly, then the activation function's non-linearity is a side effect, and like in diffusion models, noise reduction through layers might be the main role.
Reversing Transformer Notion
Attention-only transformers, which don't have MLP layers
Because they had much less success in understanding MLP layers so far (2021)
A Mathematical Framework for Transformer Circuits
Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3 , LaMDA , Codex , Meena , Gopher , and similar models. However, as these models scale, their open-endedness and high capacity creates an increasing scope for unexpected and sometimes harmful behaviors. Even years after a large model is trained, both creators and users routinely discover model capabilities – including problematic behaviors – they were previously unaware of.
https://transformer-circuits.pub/2021/framework/index.html
Induction heads - illustrated — LessWrong
Many thanks to everyone who provided helpful feedback, particularly Aryan Bhatt and Lawrence Chan! …
https://www.lesswrong.com/posts/TvrfY4c9eaGLeyDkE/induction-heads-illustrated#Q_composition
interpreting GPT: the logit lens — LessWrong
This post relates an observation I've made in my work with GPT-2, which I have not seen made elsewhere. …
https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
arxiv.org
https://arxiv.org/pdf/2405.00208
Handcrafting transformer
Neel Nanda on Twitter / X
My weak guess is that it'll also be doing rotation-y stuff in some base - regular addition is just modular addition if you correct for the rounding! And I think models find it hard to represent things precisely as a direction with highly varying norm.— Neel Nanda (@NeelNanda5) June 20, 2023
https://x.com/NeelNanda5/status/1671094151633305602