Activation Density

Creator
Creator
Seonglae Cho
Created
Created
2025 Jan 26 22:9
Editor
Edited
Edited
2025 Feb 3 11:42
Refs
Refs
fraction of nonzero activation
 
 
 
 
 
 
High Activation Density could mean either that sparsity was not properly learned, or that it is an important feature needed in various situations. In the Feature Browser, SAE features show higher feature interpretability when they have more high activation
Quantile
, which demonstrates a limitation where SAE features have low interpretability for low activations and exhibit certain skewness.
However, features with the highest
Activation Density
in the
Activation Distribution
are less interpretable, mainly because these features typically don't have high activation values in absolute terms (not quantile). A well-classified and highly interpretable SAE feature should not show density that simply decreases with activation value, but rather should show clustering at high activation levels after an initial decrease.
for quantization
 

Recommendations