AI Scaling

Creator

Creator

Alan Jo

Created

Created

2023 May 15 15:30

Editor

Editor

Alan Jo

Edited

Edited

2024 May 2 15:20

Refs

Refs

Model Generalization

More intelligence for free by scaling

recommend that training tokens should be scaled linearly with model size.

Metrics of scaling are

Computing power,

Model Training,
Statistical Model parameter

AI Scaling Notion

Emergent ability

The Bitter Lesson

Chinchilla Scaling

https://www.dwarkeshpatel.com/p/will-scaling-work

AI Scaling Methods

Depth Up-Scaling

Scaling Law (OpenAI 2020)

https://arxiv.org/pdf/2001.08361.pdf

Primate neural architecture that’s really scalable in comparison to the brains of other kinds of species, analogous to how transformers have better scaling curves than LSTMs and RNNs.

Will scaling work?

Data bottlenecks, generalization benchmarks, primate evolution, intelligence as compression, world modelers, and other considerations

Will scaling work?

https://www.dwarkeshpatel.com/p/will-scaling-work

Will scaling work?

Human brain Neuron scaling process

NCBI - WWW Error Blocked Diagnostic

Your access to the NCBI website at www.ncbi.nlm.nih.gov has been temporarily blocked due to a possible misuse/abuse situation involving your site. This is not an indication of a security issue such as a virus or attack. It could be something as simple as a run away script or learning how to better use E-utilities, http://www.ncbi.nlm.nih.gov/books/NBK25497/, for more efficient work such that your work does not impact the ability of other researchers to also use our site. To restore access and understand how to better interact with our site to avoid this in the future, please have your system administrator contact [email protected].

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2776484/

Computing is bottleneck

The Bitter Lesson

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Scaling is important

3 ways AI is scaling helpful technologies worldwide

AI I was first introduced to neural networks as an undergraduate in 1990. Back then, many people in the AI community were excited about the potential of neural networks, which were impressive, but couldn't yet accomplish important, real-world tasks. I was excited, too!

3 ways AI is scaling helpful technologies worldwide

https://blog.google/technology/ai/ways-ai-is-scaling-helpful/

3 ways AI is scaling helpful technologies worldwide

First-principles on AI scaling

How likely are we to hit a barrier?

First-principles on AI scaling

https://dynomight.net/scaling

First-principles on AI scaling

How to scale

https://openreview.net/pdf?id=iBBcRUlOAPR

Recommendations

/////