Seonglae ChoEmpirical Kronecker-Factored Approximate Curvature
Unlike K-FAC uses theoretical Expectation-based Fisher, EK-FAC approximate the value by utilizing Empirical Fisher Matrix (EFIM). This approach improve the performance of Natural Gradient Descent (NGD) in practice.
Kronecker-Factored Eigenbasis (KFE)
Kronecker factorization
By factoring matrices into Kronecker products, we can significantly reduce their dimensions, which helps manage the dimensional growth inherent in these operations.
Eigenvalue Correction
K-FAC uses the Kronecker product of two bases as approximate eigenvector bases for the full Hessian, while EK-FAC corrects eigenvalues by calculating the variance of components obtained by projecting actual pseudo-gradients onto these bases. This is called Eigenvalue Correction and enables hundreds of times faster IHPV computation compared to traditional iterative linear system solvers.
NeurIPS 2018