Graphical Model

Probabilistic graphical model (PGM), Structured Probabilistic Model

Joint Distribution that uses a graph structure to encode conditional independence assumptions. The nodes are random variables, if they have an edge between them they are directly dependent, if they do not they are

Conditionally Independent.

In a directed graph, if a node sends an arrow, it goes into the conditional part, and if it receives an arrow, it goes into the marginal part. Multiplication is done as many times as the number of nodes. When

A\rightarrow B \rightarrow C

A

and

B

are predecessors of

C

, while

A

is not a parent of

C

(only B is). In other words,

C\perp A|B

Y_i\perp Y_{pred(i) \backslash pa(i)}|Y_{pa(i)}

From factorized probability to
Directed Acyclic Graph

Independent random variables are nodes without any previous input. Random variables which are conditional on others will have an arrow from those conditional variables to themselves.

p(Y_{1:N_G}) = \prod_{i=1}^{N_G} p(Y_i \mid Y_{\text{pa}(i)})

where

N_G

denotes the number of nodes in the graph. For example when

A\rightarrow B\rightarrow C

P(A,B,C) = P(B|A) P(C|B)

Without

Collider, every directional graphical model is identical. Collider is a variable that receives multiple arrows. Fixing collider variable make variables dependent between parents.

\begin{aligned} p(x_3 \mid x_1, x_2)\, p(x_1)\, p(x_2) &= p(x_1, x_2, x_3) \\ \ne p(x_1 \mid x_3)\, p(x_2 \mid x_3)\, p(x_3) &= p(x_2 \mid x_3)\, p(x_3, x_1) = p(x_1 \mid x_3)\, p(x_3, x_2) \\ =p(x_2 \mid x_3)\, p(x_3 \mid x_1)\, p(x_1) &= p(x_2 \mid x_3)\, p(x_3, x_1) \\ =p(x_2 \mid x_3)\, p(x_3 \mid x_1)\, p(x_1) &= p(x_1 \mid x_3)\, p(x_3, x_2) \\ = p(x_1, x_2, x_3) \end{aligned}

From DAG to joint probability distribution

All of the variables have a joint probability, the nodes which do not have any arrows pointing to them are independent, the nodes which have arrows pointing to them are dependent of the nodes where the arrows originate.