In current AI systems that respond unconsciously, there will soon come a moment when Fast Weight becomes more cost-effective at increasing performance and intelligence when Reasoning Model is scaled beyond a certain level. Current AI systems that reset their memory with each inference, like an amnesiac, have clear limitations.
Way to combine Training time compute &Reasoning Model
- Temporally applying LoRA is a good candidate
- Forward Forward Algorithm without a whole memory holding is also supportive
