Seonglae ChoFIM
Def: Variance of Informant. FIM reflects Curvature of probability distribution since informant indicates the sensitivity to changes in parameter values.
Fisher (Information) Matrix
Def: Expectation for Outer Product of score function. If score function (Informant) is vector, Fisher matrix acts likeCovariance Matrix (not identical but equal when average of score function is 0 mathematically).
Empirical Fisher Matrix (EFIM)
Hessian Matrix
When the loss function is negative log-likelihood (NLL) or cross entropy, and optimization has converged using likelihood, the Hessian Matrix of loss function approximates the Fisher Information Matrix.
Fisher Information
Fisher information은 MLE를 통해 추정한 파라미터의 신뢰구간을 구할 때 등장합니다. 머신러닝과 관련해서는 KL-divergence를 Fisher Information Matrix(FIM)으로 근사할 수 있습니다.
https://velog.io/@veglog/Fisher-Information
