Matrix Multiplication

Creator

Creator

Alan Jo

Created

Created

2023 Apr 17 8:33

Editor

Editor

Alan Jo

Edited

Edited

2024 Jun 14 12:30

Refs

Refs

Dynamic Programming

Matrix chain multiplication

Anti
Commutative Property

Sigma representation

Parent methods of multiple matrix multiplication is important optimization for computing time

IBM researcher Shumuel Winograd proved 6 computation or less is impossible for 3x3 multiplication

Associative Property

Closed semiring

Matrix Multiplication Algorithms

Strassen’s Algorithm

notion image

Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]

Great minds discuss flops per watt.

Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]

https://www.thonking.ai/p/strangely-matrix-multiplications

Strangely, Matrix Multiplications on GPUs Run Faster When Given "Predictable" Data! [short]

New Breakthrough Brings Matrix Multiplication Closer to Ideal | Quanta Magazine

By eliminating a hidden inefficiency, computer scientists have come up with a new way to multiply large matrices that’s faster than ever.

New Breakthrough Brings Matrix Multiplication Closer to Ideal | Quanta Magazine

https://www.quantamagazine.org/new-breakthrough-brings-matrix-multiplication-closer-to-ideal-20240307

New Breakthrough Brings Matrix Multiplication Closer to Ideal | Quanta Magazine

LLM Matmul free

The method substitutes MatMul with ternary accumulations where the weights are only -1, 0, or +1 with GLU.

Scalable MatMul-free Language Modeling

Matrix multiplication (MatMul) typically dominates the overall computational cost of large language models (LLMs). This cost only grows as LLMs scale to larger embedding dimensions and context...

Scalable MatMul-free Language Modeling

https://arxiv.org/abs/2406.02528

Scalable MatMul-free Language Modeling

Recommendations

////////