GAE

A_{GAE}^{\pi}(s_t, a_t)=\sum_{t^{\prime}=t}^{T-1}(\gamma \lambda)^{t^{\prime}-t} \delta_{t^{\prime}},

A_{GAE}^{\pi}(s_t, a_t)=\delta_{t}+\gamma \lambda A_{GAE}^{\pi}(s_{t+1}, a_{t+1})

Sign is important in advantage because it determines training, so advantage normalization would change sign of A value.

\hat A_n^\pi(s_t,a_t) = \sum_{t'=t}^{t+n-1} r(s_{t'}, a_{t'}) + V^\pi (s_{t+n}) - V^\pi(s_t)

\hat Q_n^\pi(s_t, a_t) = \sum_{t'=t}^{t+n-1} r(s_{t'}, a_{t'}) + V^\pi(s_{t+n})

Solution: Use exponentially-weighted for future rewards average of n-step returns!

\hat A_{GAE} (s_t, a_t) = \sum_{n=1}^{\infty} w_n \hat A_n^\pi(s_t,a_t), w_n \propto \lambda^{n-1}

New hyperparameter discounting factor

\lambda

(

\lambda = 0.95

typically works well)

The lambda parameter determines a trade-off between more bias (low lambda) and more variance (high lambda).