Language Model Context

LLM is quite dumb without context

Extrapolation ability above trained context length is an important issue and this is the weak part of the

The first appearance was the hacking using

Positional Embedding. The trigonometric-based absolute positional encoding worked well by separating frequencies, but there is a problem that it does not work well unless it is data of the length often seen during training because extrapolation does not work.

Lost in the middle poured cold water on

RoPE’s prospect. It showed that these models, which expanded the context using

RoPE interpolation, referenced the beginning and end of the prompt well but did not capture the middle part well. It even stated that performance could be worse than when no data was given at all.

Gemini Google is seeking a solution with

Ring Attention.

With Reformer and Performer in mind, many efficient transformer structures have been poured out to support long context while reducing the cost of Attention's O(N^2), and there have been some promising things, but in the end, it turned out that the ordinary vanilla attention is the best as the data increases. If there is a lesson that the transformer constantly tells us since it first appeared, it is to follow the rules without tricks.

Language Model Context Notion

In-context learning

Lost in the middle

AI Knowledge Conflict

Language Model Context Usages

LM Context Extending

Contextual Compression

LM Context Optimization

Contextual Retrieval

LM Context Caching

AI Context Protocol