The pre-bias gradient uses a trick of summing pre-activation gradient across the batch
dimension before multiplying with the encoder weights. In other words, we constrain the pre-encoder bias to equal the negative of the post-decoder bias and initialize it to the geometric median of the dataset.
Each training step of a sparse autoencoder generally consists of six major computations
- the encoder forward pass
- the encoder gradient
- the decoder forward pass
- the decoder gradient
- the latent gradient
- the pre-bias gradient
Geometric median initialization
insight of Pre-bias
