Dictionary Learning

Sparse dictionary learning (SDL), Sparse Encoding

A representation learning method that aims to find a representation of input data in the form of linear combinations, which can be viewed as the opposite process of

One Hot encoding

Dictionary Learning Methods

K-Deep Simplex

Diverse Dictionary Learning

2023 SAE draft

Circuits Updates — May 2023

We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.

https://transformer-circuits.pub/2023/may-update/index.html

transformer-circuits.pub

https://transformer-circuits.pub/2023/monosemantic-features/vis/index.html

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.Browse A/1 Features →Browse All Features →

https://transformer-circuits.pub/2023/monosemantic-features/index.html

Papers with Code - Dictionary Learning

**Dictionary Learning** is an important problem in multiple areas, ranging from computational neuroscience, machine learning, to computer vision and image processing. The general goal is to find a good basis for given data. More formally, in the Dictionary Learning problem, also known as sparse coding, we are given samples of a random vector $y\in\mathbb{R}^n$, of the form $y=Ax$ where $A$ is some unknown matrix in $\mathbb{R}^{n×m}$, called dictionary, and $x$ is sampled from an unknown distribution over sparse vectors. The goal is to approximately recover the dictionary $A$. <span class="description-source">Source: [Polynomial-time tensor decompositions with sum-of-squares ](https://arxiv.org/abs/1610.01980)</span>

https://paperswithcode.com/task/dictionary-learning

Sparse dictionary learning

Sparse dictionary learning is a representation learning method which aims at finding a sparse representation of the input data in the form of a linear combination of basic elements as well as those basic elements themselves. These elements are called atoms and they compose a dictionary. Atoms in the dictionary are not required to be orthogonal, and they may be an over-complete spanning set. This problem setup also allows the dimensionality of the signals being represented to be higher than the one of the signals being observed. The above two properties lead to having seemingly redundant atoms that allow multiple representations of the same signal but also provide an improvement in sparsity and flexibility of the representation.

https://en.wikipedia.org/wiki/Sparse_dictionary_learning

superposition and interference

Superposition Without Interference? Towards Isolated Interventions...

A central premise in mechanistic interpretability is that meaningful concepts in language models are represented by linear features in activation space. For such features to support reliable...

https://arxiv.org/abs/2602.04718

Dictionary Learning

Sparse dictionary learning (SDL), Sparse Encoding

2023 SAE draft

Backlinks

Recommendations