DB-KSVD

Double-Batch K-Singular Value Decomposition

Accelerated "fast" dictionary learning algorithm optimized for large-scale LLM interpretation. Double-Batching: Both sample and dictionary updates are batch processed → optimized for GPU/parallel computing.

Motivation: Understanding whether an LLM associates "Java" with a programming language or an island requires interpreting the model's internal embeddings.

How a 20-Year-Old Algorithm Can Help Us Understand Transformer Embeddings

We introduce key optimizations to the 20 year old K-SVD algorithm, show that it can match sparse autoencoder performance for interpreting LLM embeddings, and provide theoretical insights into the feasability of the dictionary learning formulation.

https://ai.stanford.edu/blog/db-ksvd/

DB-KSVD

Double-Batch K-Singular Value Decomposition

Recommendations