JumpReLU SAE

Creator

Creator

Created

Created

2025 Mar 17 11:6

Editor

Editor

Edited

Edited

2025 May 20 14:43

Refs

Refs

Generalized Gated SAE by learning threshold by zeroing out pre-activations

+ L0 loss

JumpReLU(x, t) = \begin{cases} x, & \text{if x < exp(t)}\\ 0, & \text{otherwise} \end{cases}

JumpReLU SAE with
Unit step function

Does this mean they efficiently implemented the gating mechanism using JumpReLU activation?

\sigma(z) = JumpReLU_\theta(z) = z \odot H(z - \theta)

notion image

notion image

google jumprelu preliminary

https://arxiv.org/pdf/2404.16014

gemma scope jumprelu

https://arxiv.org/pdf/2407.14435

differentiable pre-act-loss

Circuits Updates - January 2025

We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.

Circuits Updates - January 2025

https://transformer-circuits.pub/2025/january-update/index.html

Backlinks

Sparse Autoencoder SAE Dead Feature

Recommendations

////////////