GLU function

Creator
Creator
Seonglae Cho
Created
Created
2024 Mar 30 17:50
Editor
Edited
Edited
2024 Mar 31 4:59
Refs
Refs

Gated Linear Unit

Gated mechanism usually means point-wise operation between sigmoid function to control flow (the function should not be a sigmoid but usually sigmoid is used)
Gating Mechanism
과 달리 weight는 없다
GLU(a,b)=aσ(b)GLU(a, b) = a \otimes \sigma(b)
GLU functions
 
 
 
 
 
 

Recommendations