Activation Sparsity

Creator

Creator

Seonglae Cho

Created

Created

2024 Oct 5 22:38

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Oct 5 22:42

Refs

Refs

Activation 이 sparse한 정도로 높을수록 불필요한 계산을 줄이고 모델의 효율성을 높이는 데 도움을 준다.

Superposition Hypothesis에 의해 이론이 뒷받침되며 높은 activation sparsity는

Mechanistic interpretability 에도 도움을 준다. (

Neuron SAE로 분리하기 쉽다)

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity...

Activation sparsity refers to the existence of considerable weakly-contributed elements among activation outputs. As a prevalent property of the models using the ReLU activation function,...

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity...

https://arxiv.org/abs/2402.13516

ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity...

web.stanford.edu

https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1234/final-reports/final-report-169721612.pdf

Recommendations

////////