OGPO: Sample-Efficient Finetuning of Generative Control Policies with Off-Policy Critics
Off-policy Generative Policy Optimization (OGPO) is a novel algorithm designed for sample-efficient finetuning of generative control policies (GCPs) in robotics. It leverages off-policy critic networks to maximize data reuse and propagates gradients through the full generative process using a modified PPO objective. OGPO demonstrates state-of-the-art performance on various manipulation tasks and notably can improve poorly initialized policies to high success rates without expert data in the replay buffer, while requiring minimal hyperparameter tuning.