Parameter Estimation

Creator

Creator

Created

Created

2023 Mar 23 1:36

Editor

Editor

Edited

Edited

2025 Jul 3 9:48

Refs

Refs

Approximation theory

Model Optimizer

Optimization Algorithm

Bayesian inference

Model Fitting, Fitting
Probability Distribution, Parametric Learning

Methods find the most likely parameter

\hat\theta

that explain the data

D

and boil down to

\hat{\theta} \in \argmin_\theta \mathcal{L}(\theta)

if

\Theta \subset R

,

Risk = bias^2 + variance

statistical experiment be a sample

X_1

… ,

X_n

of i.i.d. random variables in some measurable space Ω, usually Ω ⊆ ℝ

hyperparameter

\alpha

,

D

is data set

While performing MLE estimation, we update the weights through back propagation to maximize the likelihood of the data, obtaining the optimal point estimation

While performing MAP estimation, we update the weights through back propagation to maximize the posterior probability, obtaining the optimal point estimation

While performing Bayesian inference, we update the weights through back propagation to calculate the posterior probability distribution, obtaining the optimal density estimation

MLE is intuitive,

MAP is a generalized MLE with non-constant log-prior, and

ERM is a generalized form with any loss function and regularization term.

\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta} \sum_{i=1}^n \log p(y_i \mid x_i; \theta)

\begin{aligned} \hat{\theta}_{\mathrm{MAP}} &= \arg\max_{\theta}\Bigl[\sum_{i=1}^n \log p(y_i\mid x_i;\theta) + \log p(\theta)\Bigr] \\[6pt] &= \arg\min_{\theta}\Bigl[-\tfrac{1}{n}\bigl(\sum_{i=1}^n \log p(y_i\mid x_i;\theta) + \log p(\theta)\bigr)\Bigr] \\[6pt] &= \arg\min_{\theta}\Bigl[\tfrac{1}{n}\sum_{i=1}^n \underbrace{-\log p(y_i\mid x_i;\theta)}_{\ell(x_i,y_i;\theta)} + \underbrace{-\tfrac{1}{n}\log p(\theta)}_{\Omega(\theta)}\Bigr] \end{aligned}

\hat{\theta}_{\mathrm{ERM}} = \arg\min_{\theta} \Bigl[\frac{1}{n}\sum_{i=1}^n \ell\bigl(f(x_i;\theta),y_i\bigr) \;+\;\Omega(\theta)\Bigr]

Point Estimations

Parameter Estimation Notion

Back Propagation

Hebbian Meta-Learning

Forward Forward Algorithm

Variational Inference 알아보기 - MLE, MAP부터 ELBO까지

확률 분포를 근사 추정하는 기법인 Variational Inference를 이해하고 싶은 사람들을 위해, 확률 분포를 추정하는 근본적인 이유를 알려드립니다. 또한 MLE, MAP, KL divergence, ELBO 등 자주 등장하는 용어들을 설명합니다.

Variational Inference 알아보기 - MLE, MAP부터 ELBO까지

https://modulabs.co.kr/blog/variational-inference-intro/

Variational Inference 알아보기 - MLE, MAP부터 ELBO까지

Backlinks

Optimization Optimization Linear dimensionality reduction

Recommendations

///////