Q learning

Q function, policy 둘다 고려하기 어려워 No policy gradient dependency

1. Collect $\{s_i, a_i, r_i\}$ by rolling-out any policy

2. Set target $y_i = r_i + max_{a_i'}Q_\phi(s_i', a_i')$ and fit $Q_\phi(s_i,a_i)$ to $y_i$ for each data point
$Q_\phi (s,a) \leftarrow r(s,a) + max_{a'}Q_\phi(s', a')$ $\therefore$
$\phi \leftarrow argmin_\phi \sum_i ||Q_\phi (s_i, a_i) - y_i||^2$ (TD error)

3. $\pi'(s) = \argmax_aQ^\pi(s, a)$ without explicit update on $\pi$ which are defined using $Q_\phi$

Properties

• Only one Q network to learn, no high-variance policy gradient

• 하지만 하나 고정해두니 당연히 No convergence guarantees (Moving target) requires lots of tricks to make it work

• Evaluating Q-values for all possible actions is infeasible with continuous action space (Could be harder to learn than just a policy)

Value Iteration & Q Iteration