Seonglae ChoMarginalization
Forward process
which means
When we consider it as a form of Fourier Transform, we can interpret as a Linear Combination of and Gaussian Noise
Reverse process with
variational posterior
Reverse KL Divergence for model seeking and to generate more expressive data. The model learns the reverse transition by matching the variational posterior.
is the neural network we are training.
When it is normal distribution
Reparameterization trick
We compute KL of forward process and reverse process to get the Loss. To define the loss, the problem is reparameterized to predict the noise at step t rather than the structure, which empirically demonstrated to improve performance. Also, we leverage forward process's sampled reparameterized to ensure the variational posterior differentiable.
- is often fixed at 1 regardless of the step
- is a neural network instead of
ODE Solver
SDE Solver
Flow Matching with ODE → "following a map and driving in a consistent direction" while Diffusion Model with SDE → "following a map, but random wind blows at each segment". Solver: the rule that determines how often and how precisely to apply steering in reverse direction (deterministic for ODE, stochastic for SDE). Note that diffusion models can also be sampled using ODE solvers (e.g., probability flow ODE).