A Performance Comparison of Multi-Agent Reinforcement Learning Algorithms using PyMARL2

Abstract

As Multi-Agent Reinforcement Learning (MARL) develops into a promising field of artificial intelligence research and mathematical experimentation, a noticeable lack of performance comparison studies have been published, despite the standardization of training environments. The lack of a method of performance comparison could affect code reproducibility and has led to a lack of cross-examination between studies. To address this challenge, this paper conducted a performance comparison of the top-performing MARL algorithms in the StarCraft Multi-Agent Challenge (SMAC), measuring their performance under equivalent training and execution conditions called “scenarios”. This performance comparison utilized PYMARL, a training and development framework published by the authors of SMAC, in order to split models into modular units that could be interchanged without affecting environmental conditions. Each experiment trained each model three times per StarCraft faction, nine times in total per faction, and reported data over the entire process of forty-five experiments. The data resulting from these experiments was used to generate several graphs, showing the win rate of each model over time as a measure of performance and learning rate, along with several other metrics. Interestingly, as testing was performed in a lightweight environment, the resulting data implied a lessened impact on model complexity, including cases where more complex models suffered drawbacks. The compiled findings demonstrated the performance of each model in uniform experimental conditions, displaying a process by which new MARL models could be developed, trained, and tested entirely within a standardized framework. The code for this altered framework will be made publicly available and can allow future MARL work to be assured of correct results.           

Accessibility Summary:

In accordance with Title II regulations this content meets all points of exemption as Archived web content and/or Preexisting conventional electronic documents.