Actor Critic

Simultaneous learning of Policy and

State-value function

1. Sample trajectory point $\{s_i, a_i, r_i\}$

2. Update $V_\phi^{\pi_\theta}(s)$ or $Q_\phi^{\pi_\theta}(s, a)$ by like
   GAE

3. $\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_i \nabla_\theta log \pi_\theta (a_i|s_i) A_\phi^{\pi_\theta} (s_i, a_i)$

4. $\theta' \leftarrow \theta + \alpha\nabla_\theta J(\theta)$

Properties

• no need to collect full trajectories for update by sampling efficiently

• Actor-critic is sample efficient than REINFORCE

Value-based actor critic

• Unlike semi on-policy methods like PPO, this approach uses off-policy data while performing value iteration

• Instead of using GAE, it also learns a Q model, similar to algorithms like DDPG or SAC

How to update Actor

• train parameters like standard deviation of shared normal or categorical distribution