SAE Feature absorption

Feature Absorption reduces interpretability

When an SAE learns two separate features describing the same ground-truth feature, representations of that feature are split between the two learned features randomly.

Although the SAE appears to track a specific interpretable feature, in reality, it creates gaps in predictions and other unrelated latent variables absorb that feature

It seems to be an issue that occurs due to decomposing too sparsely

For example, the interpretable feature 'starts with L' is not activated under certain conditions, and instead, latent variables related to specific tokens like 'lion' absorb that direction.

It was discovered that sharing weights between the SAE encoder and decoder reduces Feature Absorption

[Paper] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders — LessWrong

This research was completed for London AI Safety Research (LASR) Labs 2024. The team was supervised by Joseph Bloom (Decode Research). Find out more…

https://www.lesswrong.com/posts/3zBsxeZzd3cvuueMJ/paper-a-is-for-absorption-studying-feature-splitting-and

Toy Models of Feature Absorption in SAEs — LessWrong

TLDR; In previous work, we found a problematic form of feature splitting called "feature absorption" when analyzing Gemma Scope SAEs. We hypothesized…

https://www.lesswrong.com/posts/kcg58WhRxFA9hv9vN/toy-models-of-feature-absorption-in-saes

SAE Feature absorption

Feature Absorption reduces interpretability

Recommendations