Scaling context-size can flexibly re-organize model representations, possibly unlocking novel capabilities. This multi-faceted examination of LLM capabilities suggests that LLMs can achieve scaling through Language Model Context scaling and LM Context Extending, demonstrating that this scaling improves connectivity and reconstruction capabilities between concepts in in-context learning and representation.
While long-context LLMs provide simplicity, their ability to accurately recall factual information can diminish. In other words, the attention mechanism struggles to manage complexity when it scales.
Language Model Context Extending Methods
