Seonglae ChoA tool of infrastructure built to enable interpretability research at Anthropic
Running remote model via garçon and Garçon client connected to a model
Hooks
Pytorch Module Hook exists. However, when the model scales beyond a single node, there's no obvious way to translate that workflow.
The basic interface to probes is that you can provide a “probe function”. Probe functions accept two arguments: a “save context” which can be used to save activations or data for later, and the tensor represented at this particular point in the model. Probe functions can return an updated tensor, which will replace the probed tensor in the computation, or can return None to use the original value. (this convention is borrowed from PyTorch hooks).
you need to create a probe, run the forward pass, and then separately retrieve the activations
rmodel.recordings() returns a dictionary indexed by probe point name.Reduction on the server
2021
Garcon
You can also watch a video covering similar content to this piece.
https://transformer-circuits.pub/2021/garcon/index.html
Infrastructure - Garcon [rough early thoughts]
See also: https://transformer-circuits.pub/2021/garcon/index.html
This video is part of a series: https://www.youtube.com/playlist?list=PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51
As an experiment, we recorded a couple videos discussing our early stage thinking on trying to reverse engineer neural networks. We made them to share our very informal thoughts with colleagues at other institutions.
Please treat these videos like talks one might give on early results at a research group meeting. Our thinking is very rough and errors are very possible. Please take all of these videos with a big grain of salt. We expect they're primarily of interest to people actively thinking about how to reverse engineer neural networks.
Our thoughts have evolved a lot since we started recording these videos.
![Infrastructure - Garcon [rough early thoughts]](https://www.notion.so/image/https%3A%2F%2Fwww.youtube.com%2Fs%2Fdesktop%2Ff45068c8%2Fimg%2Ffavicon_144x144.png?table=block&id=58f0f26e-65f1-4308-9d8e-fa8a291a96da&cache=v2)
https://www.youtube.com/watch?v=LqvCPmbg5KI&list=PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51
![Infrastructure - Garcon [rough early thoughts]](https://www.notion.so/image/https%3A%2F%2Fi.ytimg.com%2Fvi%2FLqvCPmbg5KI%2Fmaxresdefault.jpg?table=block&id=58f0f26e-65f1-4308-9d8e-fa8a291a96da&cache=v2)