Seonglae ChoIn current AI systems that respond unconsciously, there will soon come a moment when Fast Weight becomes more cost-effective at increasing performance and intelligence when Reasoning Model is scaled beyond a certain level. Current AI systems that reset their memory with each inference, like an amnesiac, have clear limitations.
Way to combine Training time compute &Reasoning Model
- Temporally applying LoRA is a good candidate
- Forward Forward Algorithm without a whole memory holding is also supportive
Geoffrey Hinton: Using Fast Weights to Store Temporary Memories
Speaker:
Geoffrey Hinton, U Toronto
https://www.youtube.com/watch?v=GLmptInTNSw
