Interpretable AI

Creator

Creator

Seonglae Cho

Created

Created

2024 May 1 1:17

Editor

Editor

Seonglae Cho

Edited

Edited

2025 Mar 4 15:26

Refs

Refs

Dimension Reduction

Interpretability

Degree to which a model can be understood in human terms

Model inspection only provides information about the model. The model might not accurately reflect the data

Interpretability paradigms offer distinct lenses for understanding neural networks: Behavioral analyzes input-output relations; Attributional quantifies individual input feature influences; Concept-based identifies high-level representations governing behavior; Mechanistic uncovers precise causal mechanisms from inputs to outputs.

https://arxiv.org/pdf/2404.14082

Interpretable AI Notion

Mechanistic interpretability

LLM Doc Transparency

Structured Latent

Interpretable Sparse Coding

Representation Similarity

Intrinsic interpretability

Interpretable Hierarchical Coding

Explainable AI Methods

Defection Probe

Faithfulness Interpretability

Minimality Interpretability

Simplicity Interpretability

Challenges

https://arxiv.org/pdf/2501.16496

200 Concrete Open Problems in Mechanistic Interpretability: Introduction — AI Alignment Forum

EDIT 19/7/24: This sequence is now two years old, and fairly out of date. I hope it's still useful for historical reasons, but I no longer recommend…

200 Concrete Open Problems in Mechanistic Interpretability: Introduction — AI Alignment Forum

https://www.alignmentforum.org/posts/LbrPTJ4fmABEdEnLf/200-concrete-open-problems-in-mechanistic-interpretability

200 Concrete Open Problems in Mechanistic Interpretability: Introduction — AI Alignment Forum

https://arxiv.org/pdf/2404.09932

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — LessWrong

Why we made this list: • * The interpretability team at Apollo Research wrapped up a few projects recently[1]. In order to decide what we’d work on…

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — LessWrong

https://www.lesswrong.com/posts/KfkpgXdgRheSRWDy8/a-list-of-45-mech-interp-project-ideas-from-apollo-research

A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team — LessWrong

Dream

Interpretability Dreams

Before diving in, it's worth making a few small remarks. Firstly, essentially all the ideas in this essay were previously articulated, but buried in previous papers. Our goal is just to surface those implicit visions, largely by quoting relevant parts. Secondly, it's important to note that everything in this essay is almost definitionally extremely speculative and uncertain. It's far from clear that any of it will ultimately be possible. Finally, since the goal of this essay is to lay out our personal vision of what's inspiring to us, it may come across as a bit grandiose – we hope that it can be understood as simply trying to communicate subjective excitement in an open way.

Interpretability Dreams

https://transformer-circuits.pub/2023/interpretability-dreams/index.html

Recommendations

///////