Fisher Information Matrix

FIM

Def:

Variance of

Informant. FIM reflects

Curvature of probability distribution since informant indicates the sensitivity to changes in parameter values.

Fisher (Information) Matrix

Def: Expectation for

Outer Product of score function. If score function (

Informant) is vector, Fisher matrix acts like

Covariance Matrix (not identical but equal when average of score function is 0 mathematically).

F = \mathbb{E}[s(\theta)s(\theta)^T]

Empirical Fisher Matrix (EFIM)

\hat{F} = \frac{1}{N}s(\theta_i)s(\theta_i)^T

Hessian Matrix

When the loss function is negative log-likelihood (NLL) or cross entropy, and optimization has converged using likelihood, the Hessian Matrix of loss function

H \approx F

approximates the Fisher Information Matrix.

arxiv.org

https://arxiv.org/pdf/2411.12580

Fisher Information

Fisher information은 MLE를 통해 추정한 파라미터의 신뢰구간을 구할 때 등장합니다. 머신러닝과 관련해서는 KL-divergence를 Fisher Information Matrix(FIM)으로 근사할 수 있습니다.

https://velog.io/@veglog/Fisher-Information

Fisher Information Matrix

FIM

Fisher (Information) Matrix

Empirical Fisher Matrix (EFIM)

Hessian Matrix

Backlinks

Recommendations