LLM Memory (Overfitting)
To ultimately become like humans, knowledge needs to be distributed in vector space, but since performing weight training at every moment is too inefficient, we will likely need a separate storage system like Vector Database or Fast Weight.
AI Memory Mechanism is distinguished from AI Reasoning mechanism. While these two processes support each other, they remain separate. Through neuron activation classification probes, we can classify memorized tokens from non-memorized tokens with over 99.9% accuracy. Additionally, it's possible to effectively remove the memory mechanism by suppressing neuron activation patterns that depend on memorized data. Research has discovered that specific neurons (e.g., neuron 1668) play a crucial role in representing the model's confidence.
While Attention Mechanism implemented and mimiced Working memory(Short Term Memory) effectively, Long Term Memory implementation is not incorporated.
KV matching is a memory write operation while query is a read operation
AI Memory Implementation
AI Memory Models
AI Memory Tools
Working memory = Language Model Context
However, while LLMs use a fixed window of tokens, the human brain goes a step further by continuously updating a summary of past information to integrate longer-term context.
Mechanistic Analysis
However, while LLMs use a fixed window of tokens, the human brain goes a step further by continuously updating a summary of past information to integrate longer-term context.
