Monarch Mixer

Creator

Creator

Created

Created

2024 Jan 17 14:56

Editor

Editor

Edited

Edited

2024 Jan 17 14:59

Refs

Refs

M2

Monarch 매트릭스를 사용하여 시퀀스와 모델 차원에서의 혼합 작업을 수행합니다. 이는 전통적인 어텐션 메커니즘을 대체

Monarch matrices are a sub-quadratic primitive (you can compute them in

O(N^{3/2})

that are also hardware-efficient and expressive

notion image

Monarch Mixer Notion

Dimension Mixer

Long-Context Retrieval Models with Monarch Mixer

Long-Context Retrieval Models with Monarch Mixer

https://hazyresearch.stanford.edu/blog/2024-01-11-m2-bert-retrieval

Long-Context Retrieval Models with Monarch Mixer

https://arxiv.org/pdf/2310.12109.pdf

Recommendations

//////////