Texonom
Texonom/
Engineering
Engineering/Data Engineering/Artificial Intelligence/AI Industry/
AI Scaling
Search

AI Scaling

Creator
Creator
Seonglae ChoSeonglae Cho
Created
Created
2023 May 15 15:30
Editor
Editor
Seonglae ChoSeonglae Cho
Edited
Edited
2024 Dec 3 21:4
Refs
Refs
Scalability
Model Generalization
Economies of scale

More intelligence for free by scaling

Andrej Karpathy said that It will be surprisingly small since the current models are wasting a ton of capacity remembering stuff that does not matter. There will be a cognitive core (math, physics, computing, predicting) like human brain similar to brain’s layered structure.
Recommended that training tokens should be scaled linearly with model size. The constraints on scaling test-time compute approach differ substantially from those of LLM pretraining.
Metrics of scaling are
Computing
Computing power,
Model Training,
Statistical Model parameter
AI Scaling Notion
Emergent ability
The Bitter Lesson
Chinchilla Scaling
Scaling Law
https://www.dwarkeshpatel.com/p/will-scaling-work
AI Scaling Methods
Depth Up-Scaling
Model Merging
 
 
 
 

Scaling Law (OpenAI 2020)

arxiv.org
https://arxiv.org/pdf/2001.08361.pdf
Primate neural architecture that’s really scalable in comparison to the brains of other kinds of species, analogous to how transformers have better scaling curves than LSTMs and RNNs.
Will scaling work?
Data bottlenecks, generalization benchmarks, primate evolution, intelligence as compression, world modelers, and other considerations
Will scaling work?
https://www.dwarkeshpatel.com/p/will-scaling-work
Will scaling work?

Human brain Neuron scaling process

NCBI - WWW Error Blocked Diagnostic
Your access to the NCBI website at www.ncbi.nlm.nih.gov has been temporarily blocked due to a possible misuse/abuse situation involving your site. This is not an indication of a security issue such as a virus or attack. It could be something as simple as a run away script or learning how to better use E-utilities, http://www.ncbi.nlm.nih.gov/books/NBK25497/, for more efficient work such that your work does not impact the ability of other researchers to also use our site. To restore access and understand how to better interact with our site to avoid this in the future, please have your system administrator contact [email protected].
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2776484/

Computing is bottleneck

The Bitter Lesson
http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Scaling is important

3 ways AI is scaling helpful technologies worldwide
AI I was first introduced to neural networks as an undergraduate in 1990. Back then, many people in the AI community were excited about the potential of neural networks, which were impressive, but couldn't yet accomplish important, real-world tasks. I was excited, too!
3 ways AI is scaling helpful technologies worldwide
https://blog.google/technology/ai/ways-ai-is-scaling-helpful/
3 ways AI is scaling helpful technologies worldwide
First-principles on AI scaling
How likely are we to hit a barrier?
First-principles on AI scaling
https://dynomight.net/scaling
First-principles on AI scaling

How to scale

openreview.net
https://openreview.net/pdf?id=iBBcRUlOAPR
 
 

Recommendations

Texonom
Texonom/
Engineering
Engineering/Data Engineering/Artificial Intelligence/AI Industry/
AI Scaling