Core Skill Decomposition of Complex Wargames with Reinforcement Learning

In recent years, Reinforcement Learning (RL) agents were able to solve the most challenging games to the extent that they competed and even surpassed the most successful human players. This suggests that RL methods are well suited for wargames where the complexity arises from very long decision horizons, sparse rewards, and large action spaces. Due to the complex nature of wargames, even with RL, convergence to a near-optimum solution requires an immense amount of experience and makes the solution sample inefficient. In order to address the inefficiency, we propose to divide the game into simpler sub-games, where each sub-game covers a core skill of the game. These sub-games have shorter decision horizons and smaller action spaces compared to the main game. We employ a curriculum learning setting with a hierarchical control structure, where the curriculum consists of simpler sub-games. We choose StarCraft II as our test bench as it possesses the common features of wargames and it has been extensively used in wargame scenarios. We empirically show that our proposed hierarchical architecture is able to solve a complex wargame environment based on StarCraft II game whereas the non-hierarchical agent fails to solve. We further observed that a set of core skills is sufficient to achieve near-optimal scores, and a larger set of skills beyond the core skills only marginally improves the performance.