Narrowly defined AI
Neural Network is a differentiable computer.
The rows of the weight matrix before the activation function can be thought of as directions in the embedding space, and that means activation of each neuron tells you how much a given vector aligns with some specific direction. The columns of the weight matrix after the activation function tell you what will be added to the result if that neuron is active.
Data Representation + Data Classification + Algorithm
Neural network layer common architecture
- layer normalization
- main logic like Attention Mechanism or mlp with activation function
- projection to original size (even if it is same dimension size)
- residual connection
Neural Network Notion
