Seonglae ChoEmpirical Kronecker-Factored Approximate Curvature
Unlike K-FAC uses theoretical Expectation-based Fisher, EK-FAC approximate the value by utilizing Empirical Fisher Matrix (EFIM). This approach improve the performance of Natural Gradient Descent (NGD) in practice.
Kronecker-Factored Eigenbasis (KFE)
Eigenvalue Correction
K-FAC uses the Kronecker product of two bases as approximate eigenvector bases for the full Hessian, while EK-FAC corrects eigenvalues by calculating the variance of components obtained by projecting actual pseudo-gradients onto these bases. This is called Eigenvalue Correction and enables hundreds of times faster computation compared to traditional iterative linear system solvers.
NeurIPS 2018
arxiv.org
https://arxiv.org/pdf/1806.03884