An improved reinforcement learning path planning algorithm for AUVs operating in unknown and unstructured environments

Abstract

Effective path planning is vital for Autonomous Underwater Vehicles (AUVs) for performing diverse roles, including rescue efforts and logistics operations. In recent times, dynamic path planning has emerged as a prominent research area, enabling AUVs to navigate without the need for previous static information. Researchers are leveraging deep reinforcement learning (DRL), as another trending field of study, to tackle the challenges associated with dynamic path planning. In this study a dynamic sensing and collision avoidance approach is proposed to navigate in an entirely unknown environment with unstructured obstacles. Firstly, we employ Deep Q-Networks (DQN), a distinct domain within DRL, to address the challenge of path planning in a dynamic environment. Initially, established methodologies were implemented such as Double Deep Q-Networks (DDQN) and Dueling Double Deep Q-Networks (D3QN) to develop a model capable of navigating an AUV through environments containing both static and dynamic obstacles. Nevertheless, the previously mentioned techniques exhibit restricted generalization abilities in complex, high-dimensional state environments, frequently leading to inadequate performance in dynamic situations, thus making the more adaptable I-DQN method essential. Subsequently, Improved DQN (I-DQN) is introduced, an enhancement of the original DQN, aimed at further refining the DRL model's performance. Test within a real-time setting is carried out to evaluate the effectiveness of the DRL models against randomly generated starting and destination points. The results obtained indicate that the I-DQN approach outperforms DQN, DDQN, and D3QN regarding the path efficiency and the travel duration to reach the destination while avoiding collisions. The stability of the I-DQN algorithm is improved by Prioritized Experience Replay, which carefully selects significant transitions according to their temporal-difference errors, facilitating effective and reliable learning in intricate AUV navigation situations. The proposed strategy can be utilized to improve underwater search and rescue missions in inundated urban regions or areas affected by disasters. The AUVs can maneuver through intricate and unfamiliar underwater settings, skilfully dodging debris and obstructions, to swiftly and efficiently find survivors or evaluate structural damage.