ES-SAC: A hybrid evolution strategy and reinforcement learning approach for humanoid locomotion control

Abstract

State-of-the-art deep reinforcement learning (DRL) techniques such as Soft Actor-Critic (SAC), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Deep Deterministic Policy Gradient (DDPG) demonstrate promising results in developing control strategies. In this study, we propose ES-SAC, a hybrid learning framework that integrates Evolutionary Strategy (ES) with the SAC algorithm to enhance humanoid robot locomotion control. ES-SAC leverages the global search capabilities of evolutionary algorithms and the sample efficiency and convergence properties of DRL. The performance of the ES-SAC agent was evaluated on a bipedal robot simulation and compared to other hybrid methods employing deterministic agents, including ES-TD3 and ES-DDPG. The ES-SAC agent exhibited superior average reward performance and a more stable learning process. In contrast, the ES-TD3 agent achieved faster course completion but exhibited control instabilities. This study also highlights the importance of physical and behavioral metrics – such as torque efficiency, horizontal and vertical deflection, and Q₀ values – in assessing the reliability of DRL-based locomotion control. Our findings suggest that relying solely on cumulative reward for evaluation can be misleading, underscoring the need for a more comprehensive analysis in future research.