Seonglae ChoSparse dictionary learning (SDL), Sparse Encoding
A representation learning method that aims to find a representation of input data in the form of linear combinations, which can be viewed as the opposite process of One Hot encoding
Dictionary Learning Methods
Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.Browse A/1 Features →Browse All Features →
https://transformer-circuits.pub/2023/monosemantic-features/index.html
Papers with Code - Dictionary Learning
**Dictionary Learning** is an important problem in multiple areas, ranging from computational neuroscience, machine learning, to computer vision and image processing. The general goal is to find a good basis for given data. More formally, in the Dictionary Learning problem, also known as sparse coding, we are given samples of a random vector $y\in\mathbb{R}^n$, of the form $y=Ax$ where $A$ is some unknown matrix in $\mathbb{R}^{n×m}$, called dictionary, and $x$ is sampled from an unknown distribution over sparse vectors. The goal is to approximately recover the dictionary $A$. <span class="description-source">Source: [Polynomial-time tensor decompositions with sum-of-squares ](https://arxiv.org/abs/1610.01980)</span>
https://paperswithcode.com/task/dictionary-learning
Sparse dictionary learning
Sparse dictionary learning is a representation learning method which aims at finding a sparse representation of the input data in the form of a linear combination of basic elements as well as those basic elements themselves. These elements are called atoms and they compose a dictionary. Atoms in the dictionary are not required to be orthogonal, and they may be an over-complete spanning set. This problem setup also allows the dimensionality of the signals being represented to be higher than the one of the signals being observed. The above two properties lead to having seemingly redundant atoms that allow multiple representations of the same signal but also provide an improvement in sparsity and flexibility of the representation.
https://en.wikipedia.org/wiki/Sparse_dictionary_learning
