AI Neuron Activation

Creator

Creator

Seonglae Cho

Created

Created

2023 Nov 12 18:44

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Oct 7 13:39

Refs

Refs

Monosemanticity

The first step in that program is to identify the correct components to analyze.

Unfortunately, the most natural computational unit of the neural network – the neuron itself turns out not to be a natural unit for human understanding. This is because many neurons are polysemantic(

Superposition). Superposition can arise naturally during the course of neural network training if the set of features useful to a model are sparse in the training data.

https://openai.com/research/language-models-can-explain-neurons-in-language-models

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.Browse A/1 Features →Browse All Features →

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

https://transformer-circuits.pub/2023/monosemantic-features/index.html

Language models can explain neurons in language models

We use GPT-4 to automatically write explanations for the behavior of neurons in large language models and to score those explanations. We release a dataset of these (imperfect) explanations and scores for every neuron in GPT-2.

Language models can explain neurons in language models

https://openai.com/research/language-models-can-explain-neurons-in-language-models

Language models can explain neurons in language models

Runtime Monitoring Neuron Activation Patterns

For using neural networks in safety critical domains, it is important to know if a decision made by a neural network is supported by prior similarities in training. We propose runtime neuron...

Runtime Monitoring Neuron Activation Patterns

https://arxiv.org/abs/1809.06573

Runtime Monitoring Neuron Activation Patterns

Safety features

High-Level Actions

Planning

Social Reasoning

Personas

Circuits Updates - July 2023

We report a number of developing ideas on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. Some of these are emerging strands of research where we expect to publish more on in the coming months. Others are minor points we wish to share, since we're unlikely to ever write a paper about them.

Circuits Updates - July 2023

https://transformer-circuits.pub/2023/july-update/index.html#safety-features

Recommendations

//////////