Seonglae Cho20 tokens per parameter
Previous models are undertrained (5 tokens per parameter) where represents the compute budget and represents the number of model parameters.
Optimal amount of data a model of a given size should be trained on
Larger models require more data
- Brain size corresponds to model size
- Learning period before adulthood corresponds to training data size
According to the Chinchilla paper, the human brain is most efficient with millions of years of learning
However, in nature, due to external factors, humans and other living organisms face a trade-off between resources used for the brain and resources needed for survival. The expected future gains from learning decrease exponentially while surviving during the learning period, so they cannot have a sufficient training period.
In modern society, the expected return on learning period is higher (though the correlation between intelligence and income is still low from top performers to average), and the burden of learning has decreased compared to natural conditions, so the learning period is increasing.
For robots, this selective pressure on intelligence does not apply, so only the linear computational cost of learning is incurred.
from DeepMind
An empirical analysis of compute-optimal large language model training
We investigate the optimal model and dataset size for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training \nummodels language models ranging from 70 million to 10 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the training dataset size should be scaled equally: for every doubling of model size the training dataset size should also be doubled. We test this hypothesis by training a more compute-optimal model, \Chinchilla, using the same compute budget as \gopher but with 70B parameters and 4$\times$ more data. \chinchilla uniformly and significantly outperforms \Gopher, GPT-3, Jurassic-1, and \mtnlg on a large range of downstream evaluation tasks. As a highlight, \chinchilla reaches an average accuracy of 67.5\% on the MMLU benchmark, over a 7\% improvement over \gopher.
https://www.deepmind.com/publications/an-empirical-analysis-of-compute-optimal-large-language-model-training
컴퓨팅의 의미, 인간의 지능은 특별한가 - Carl Shulman
일리야 수츠케버는 GPU는 새로운 비트코인이다고 말했고, 일론 머스크는 최근 GPU를 구하는 것이 마약보다 어렵다고 말했습니다. 왜 모든 기업들이 ASIC 칩을 준비하고, 모두가 GPU를 주문하며 컴퓨팅 규모를 키우는 것일까요?
정말 미래의 모델 성능은 실제 사회에 투입이 가능한 지점까지 갈 수 있을까요? 인간만이 가지는 단계는 없는 것일까요. 만약 그렇다면, 인간의 지능은 특별한 것일까요.
지능이 인간을 행성의 지배자로 만들어줄만큼 대단한 것이었다면, 왜 다른 동물들은 더 똑똑한 방향으로 진화하지 못한 것일까요. 왜 인간은 더욱 똑똑해지지 못한 것일까요.
이 모든 이야기를 담은 아주 흥미로운 인터뷰를 Dwarkesh Patel이 진행하는 The Lunar Society에서 담아냈습니다.
Carl Shulman은 옥스포드의 Future of Humanity Institute 연구원이자 Open Philanthropy Project의 어드바이저이며 이전에는 기계지능연구소(MIRI, Machine Intelligence Research Institute)에 몸담았던 사람입니다. 닉 보스트롬과 Propositions Concerning Digital Minds and Society라는, 고도의 AI가 사회에 통합되는 과정에 대한 논문을 쓰기도 한 연구자입니다.
우리의 관점을 바꿔줄만큼 흥미로운 인터뷰, 함께 들어볼까요?
https://www.youtube.com/watch?v=_kRg-ZP1vQc&t=6533s
https://www.youtube.com/@UCXl4i9dYBrFOabk0xGmbkRA
https://www.youtube.com/watch?v=nbai4z06Z4w
Neural scaling law
In machine learning, a neural scaling law is a scaling law relating parameters of a family of neural networks.[1][2]
https://en.wikipedia.org/wiki/Neural_scaling_law
Training Compute-Optimal Large Language Models (Chinchilla Scaling )