MoE

Creator
Creator
Seonglae ChoSeonglae Cho
Created
Created
2023 Apr 12 14:20
Editor
Edited
Edited
2024 Nov 22 20:57

Mixture-of-Experts

Mixture-of-Experts models improves efficiency by activating a small subset of model weights for a given input, decoupling model size from inference efficiency.
MoEs have seen great success in LLMs. In a nutshell, MoEs are pre-trained faster, and have a faster inference, but require more memory and face challenges in fine-tuning.
MoE Notion
 
 
 

Structure

notion image
1991
 
 

Recommendations