Seonglae ChoLanguage Modeling in a Sentence Representation Space
Since it utilizes the SONAR embedding space (frozen encoder), it is superficially independent of language and modality (since SONAR is a multimodal encoder). While it is fundamentally a Transformer, as a Diffusion-based LCM, it learns the conditional probability distribution of the next sentence embedding using a diffusion model.
Limitation
Since sentence embedding predictions involve too many possible sentence combinations, more training data and sophisticated modeling are needed to generate appropriate next sentences. This presents a limitation that requires expansion to both smaller and larger units beyond the sentence level. Additionally, it shares SONAR's limitations.
Large Concept Models: Language Modeling in a Sentence Representation Space | Research - AI at Meta
LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of...
https://ai.meta.com/research/publications/large-concept-models-language-modeling-in-a-sentence-representation-space/