Seong-lae ChoArtificial Neural Network
A statistical learning algorithm inspired by biological neural networks in machine learning and cognitive science
All Neural Network and Model Optimizer can be viewed as Associative Memory that compresses context flow.
Narrowly defined AI
Neural Network is a differentiable computer.
The rows of the weight matrix before the activation function can be thought of as directions in the embedding space, and that means activation of each neuron tells you how much a given vector aligns with some specific direction. The columns of the weight matrix after the activation function tell you what will be added to the result if that neuron is active.
Data Representation + Data Classification + Algorithm
Neural network layer common architecture
- layer normalization
- main logic like Attention Mechanism or mlp with activation function
- projection to original size (even if it is same dimension size)
- residual connection
Neural Network Notion
Visualization
Tensorflow — Neural Network Playground
Tinker with a real neural network right here in your browser.
https://playground.tensorflow.org/
Zero to Hero
Neural Networks: Zero To Hero
A course by Andrej Karpathy on building neural networks, from scratch, in code. We start with the basics of backpropagation and build up to modern deep neural networks, like GPT. In my opinion language models are an excellent place to learn deep learning, even if your intention is to eventually go to other areas like computer vision because most of what you learn will be immediately transferable.
https://karpathy.ai/zero-to-hero.html