Research Area:  Machine Learning

Abstract:

Reinforcement learning (RL) has achieved remarkable advancements in navigation tasks in recent years. However, tackling multi-goal navigation tasks with sparse rewards remains a complex and challenging problem due to the long-sequence decision-making involved. Such multi-goal navigation tasks inherently incorporate a hybrid action space, where the robot needs to select a navigation endpoint first before executing primitive actions. To address the problem of multi-goal navigation with sparse rewards, we introduce a novel hierarchical RL framework named Hierarchical RL with Multi-Goal (HRL-MG). The main idea of HRL-MG is to divide and conquer the hybrid action space, splitting long-sequence decisions into short-sequence decisions. The HRL-MG framework is composed of two main modules: a selector and an actuator. The selector employs a temporal abstraction hierarchical architecture designed to specify a desired end goal based on the discrete action space. Conversely, the actuator utilizes a continuous goal-oriented hierarchical architecture developed to enact continuous action sequences to reach the desired end goal specified by the selector. In addition, we incorporate a dynamic goal detection mechanism, grounded in hindsight experience replay, to mitigate the challenges posed by sparse reward landscapes. We validated the algorithm’s efficacy on both the discrete environment Maze_2D and the continuous robotic environment MuJoCo Ant. The results indicate that HRL-MG significantly outperforms other methods in multi-goal navigation tasks with sparse rewards.

Keywords:  

Author(s) Name:  Jiangyue Yan, Biao Luo, Xiaodong Xu

Publisher name:  Springer

DOI:  10.1007/s10462-024-10794-3

Volume Information:  Volume 57, article number 156, (2024)

Paper Link:   https://link.springer.com/article/10.1007/s10462-024-10794-3