Matrix Multiplication

Creator
Creator
Seonglae Cho
Created
Created
2023 Apr 17 8:33
Editor
Edited
Edited
2024 Jun 14 12:30

Anti
Commutative Property

ABBAAB \neq BA

Sigma representation

A(Bu)=(AB)u=w=CuA(Bu) = (AB)u = w = Cuwi=kAikrBkrur=r(kAikBkr)ur=r(Cir)urw_i = \sum_k{A_{ik}\sum_r{B_{kr}u_r}} = \sum_r{(\sum_k{A_{ik}B_{kr}})u_r} = \sum_r{(C_{ir})u_r}
Parent methods of multiple matrix multiplication is important optimization for computing time
IBM researcher Shumuel Winograd proved 6 computation or less is impossible for 3x3 multiplication
Matrix Multiplication Algorithms
 
notion image
 
 
 

LLM Matmul free

The method substitutes MatMul with ternary accumulations where the weights are only -1, 0, or +1 with GLU.
 
 

Recommendations