Seonglae ChoFIM
Def: Variance of Informant. FIM reflects Curvature of probability distribution since informant indicates the sensitivity to changes in parameter values.
Fisher (Information) Matrix
Def: Expectation for Outer Product of score function. If score function (Informant) is vector, Fisher matrix acts likeCovariance Matrix (not identical but equal when average of score function is 0 mathematically).
Empirical Fisher Matrix (EFIM)
Hessian Matrix
When the loss function is negative log-likelihood (NLL) or cross entropy, and optimization has converged using likelihood, the Hessian Matrix of loss function approximates the Fisher Information Matrix.
arxiv.org
https://arxiv.org/pdf/2411.12580
Fisher Information
Fisher information은 MLE를 통해 추정한 파라미터의 신뢰구간을 구할 때 등장합니다. 머신러닝과 관련해서는 KL-divergence를 Fisher Information Matrix(FIM)으로 근사할 수 있습니다.
https://velog.io/@veglog/Fisher-Information
