Alan JoKolmogorov-Arnold Networks
Effectively prevent Catastrophic forgetting and alternative of replacing MLP of transformer
complex activation functions demand more computational resources.
It turns out, that you can write Kolmogorov-Arnold Network as an MLP, with some repeats and shift before ReLU.
The performance issue of the original implementation comes from expanding all intermediate variables to perform the different activation functions. This repository implements a solution to this efficiency gap by formulating all activation functions as a linear combination of a fixed set of basis functions.
![[D] Kolmogorov-Arnold Network is just an MLP](https://www.notion.so/image/https%3A%2F%2Fwww.redditstatic.com%2Fshreddit%2Fassets%2Ffavicon%2F192x192.png?table=block&id=eb94ccab-a3e0-44a1-8fb8-908a0694b32b&cache=v2)