Seonglae ChoSTE
for non-differentiable function’s back propagation
In forward propagation, non-differentiable functions are used, but during backpropagation, their derivatives are approximated simply (often like an identity function) to bypass non-differentiable parts and update the weights of the entire model
Circuits Updates - January 2025
We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.
https://transformer-circuits.pub/2025/january-update/index.html