my-first-blog

Hello World!

This is my first blog, and I am excited to share my thoughts and experiences with you. Stay tuned for more updates!

Policy Gradient Formula

\[\nabla_\theta J(\theta) = \mathbb{E} \left[ \nabla_\theta \log \pi(a|s,\theta) \cdot Q(s,a) \right].\]

\(\textit{Proof}\).

\[\begin{aligned} \nabla_\theta J(\theta) &= \nabla_\theta \mathbb{E}_{\tau \sim \pi_\theta} [R(\tau)] \\ &= \nabla_\theta \int_\tau P(\tau|\theta) R(\tau) d\tau \\ &= \int_\tau \nabla_\theta P(\tau|\theta) R(\tau) d\tau \\ &= \int_\tau P(\tau|\theta) \nabla_\theta \log P(\tau|\theta) R(\tau) d\tau \\ &= \mathbb{E}_{\tau \sim \pi_\theta} [\nabla_\theta \log P(\tau|\theta) R(\tau)] \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) R(\tau) \right] \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) \sum_{t'=t}^{T-1} r(s_{t'},a_{t'}) \right] \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) Q(s_t,a_t) \right] \end{aligned}\]

The bias of the policy gradient estimator can be reduced by using a baseline function \(b(s_t)\), which does not depend on the action \(a_t\). Proof:

\[\begin{aligned} \nabla_\theta J(\theta) &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) (Q(s_t,a_t) - b(s_t)) \right] \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) Q(s_t,a_t) \right] - \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) b(s_t) \right] \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) Q(s_t,a_t) \right] - \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) \right] b(s_t) \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) Q(s_t,a_t) \right] - 0 \\ &= \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi(a_t|s_t,\theta) Q(s_t,a_t) \right] \end{aligned}\]