Quantization SAE

Creator

Creator

Seonglae Cho

Created

Created

2024 Oct 24 0:3

Editor

Editor

Seonglae Cho

Edited

Edited

2024 Oct 31 9:46

Refs

Refs

Codebook feature
VQ-VAE

Residual Vector Quantization bottleneck 양자화된 벡터가 더 유의미한 정보만을 포함하도록

https://arxiv.org/pdf/2310.17230

RQAE
Residual Vector Quantization
VQ-VAE

RQAE learns features hierarchically

RQAE has much higher capacity than SAEs

RQAE has much higher capacity than SAEs

Harish Kamath | RQAE - Hierarchical LLM Representations

Harish Kamath's personal webpage.

https://hkamath.me/blog/2024/rqae/

Recommendations

///////////