Ship course-keeping in waves using sample-efficient reinforcement learning

Maintaining a steady course in waves is important for ships for various reasons such as safety, fuel efficiency, and comfort. This has commonly been addressed by adopting conventional control algorithms. Reinforcement learning (RL) methods, on the other hand, have demonstrated successful performance in a wide range of control problems. In this work, the performance of two RL agents (model-free and model-based) in comparison to a linear-quadratic regulator (LQR) is investigated in a numerical environment. The model-free RL agent performed better than the LQR with respect to keeping its course and minimizing the rudder usage. By applying model-based RL, the low sample efficiency and consequent long training times that typically complicate model-free RL were mitigated. As a result, the training time of the course-keeping agent was reduced by more than an order of magnitude. Moreover, the model-based agent learned to exclusively react to the low-frequency yaw motion while ignoring the first-order wave disturbances, thereby reducing the rudder usage considerably.