Seonglae ChoDouble-Batch K-Singular Value Decomposition
Accelerated "fast" dictionary learning algorithm optimized for large-scale LLM interpretation. Double-Batching: Both sample and dictionary updates are batch processed → optimized for GPU/parallel computing.
Motivation: Understanding whether an LLM associates "Java" with a programming language or an island requires interpreting the model's internal embeddings.
How a 20-Year-Old Algorithm Can Help Us Understand Transformer Embeddings
We introduce key optimizations to the 20 year old K-SVD algorithm, show that it can match sparse autoencoder performance for interpreting LLM embeddings, and provide theoretical insights into the feasability of the dictionary learning formulation.
https://ai.stanford.edu/blog/db-ksvd/
