Deep excavation wall design using reinforcement learning

Abstract

This study investigates the application of reinforcement learning (RL) for obtaining near-optimal designs of diaphragm walls in geotechnical engineering. A physics-based numerical environment is developed to simulate soil–structure interaction, relying on a Winkler spring formulation with pressure-dependent soil springs to approximate the nonlinear response of the ground. This modelling framework allows the agent to evaluate candidate designs through physically meaningful structural responses rather than surrogate performance indicators. To reflect realistic engineering practice, a specifically designed action space and reward function are formulated, incorporating both discrete design decisions and continuous geometric parameters. During training, the agent iteratively proposes a design configuration, observes the response computed by the physical simulator, and updates its policy based on the resulting reward signal. Several common RL algorithms are investigated, including Proximal Policy Optimization (PPO), REINFORCE, and the Parameterized Deep Q-Network (P-DQN), enabling a comparative assessment of policy-based and hybrid value based approaches for this task. The algorithms are evaluated in terms of learning stability, convergence behaviour, and the quality of the resulting design solutions. The results demonstrate the potential of RL-based methods to explore complex design spaces efficiently while respecting physical constraints, highlighting their suitability for supporting automated or decision-assisted design of diaphragm walls.