Pretraining

The process of artificial neural networks extracting features from data and abstracting separation in each neuron.

• Data efficiency matters and there is an optimism. Algorithmic changes stack so well.
  Sample efficiency almost with human level learning is still far away.

• Semi-synchronous scaling might work with 10+ million GPUs in the future since not all parts of the brain necessarily need to communicate with each other.

• For the scaling law, the problem is that extending the tail of lower probability requires 10x more computation since relevant concepts appear sparsely in the long tail

How training process and loss value is related to neural network’s ability