Seonglae ChoGeneralized Gated SAE by learning threshold by zeroing out pre-activations
+ L0 loss
JumpReLU SAE with Unit step function
Does this mean they efficiently implemented the gating mechanism using JumpReLU activation?
google jumprelu preliminary
gemma scope jumprelu
differentiable pre-act-loss
Circuits Updates - January 2025
We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.
https://transformer-circuits.pub/2025/january-update/index.html