Needle retrieval task

Needle in a Haystack

• a 'needle' — a fact or statement — is embedded within a 'haystack' — a lengthy, detailed context.

• the model is tasked to retrieve this 'needle' from its memory.

• model's retrieval precision is measured by varying the the position of the 'needle' and the size of the 'haystack'

NIAH (S-NIAH)

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent...

We introduce Michelangelo: a minimal, synthetic, and unleaked long-context reasoning evaluation for large language models which is also easy to automatically score. This evaluation is derived via...

https://arxiv.org/abs/2409.12640