Inference-Time Decomposition of Activations
Due to the Linear representation hypothesis, a greedy algorithm is possible. It's applied with Matching pursuit from the compressed sensing algorithm field and has much room for improvement. While it could be interesting if calculations become efficient for large models or cross-model scenarios, looking at the appendix shows there aren't many good features relative to the model size.
Maintains over 90% reconstruction performance compared to SAE, while achieving similar performance in automated interpretation and linear probing tasks with 100-1000x faster learning speed. Offers an alternative to SAE for large-scale and repetitive analysis tasks that would otherwise be computationally burdensome.
