Seonglae ChoGood
Lightweight text embedding model: 308M parameters, runs on-device with less than 200MB RAM when quantized. Achieves top performance among sub-500M models on MTEB (multilingual, English, and code).
- Gemma 3-based encoder-decoder initialization. Gemma 3 is a decoder-only LLM. It's further trained as an encoder-decoder structure, then only the encoder is extracted and used as the initial weights for the embedding model.
- Large model distillation from Gemini Embedding
- Spread-Out Regularization: a constraint that prevents embeddings from collapsing to a single point or direction.
- Model Souping: averaging weights from multiple models trained on different data mixes. "Merging a brain good at retrieval + a brain good at classification 50/50"
- MRL (Matryoshka Representation Learning): supports variable embedding dimensions with MRL. Matryoshka Embedding
Huggingface
How to use
V2 multimodal
- text only 270
- multimodal 800
