计算机科学与探索

• 学术研究 •    下一篇

基于隐式通信的值分解多智能体强化学习

邓亚男,王秋红,李俊杰,顾晶晶   

  1. 南京航空航天大学 计算机科学与技术学院,南京210000

Value Function Factorization for Multi-Agent Reinforcement Learning based on Implicit Communication

DENG Yanan,WANG Qiuhong,LI Junjie,GU Jingjing   

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 210000, China

摘要: 在多智能体系统中,智能体通常只能观察到部分状态信息,导致每个智能体在做决策时缺乏对其他智能体行为和环境动态的完整理解,进而增加了协作的难度。虽然基于值函数分解的多智能体强化学习方法对解决局部可观测性问题有一定的优势,但由于状态-动作空间维度高、模型结构复杂等问题,多智能体系统中仍然存在着协作不确定性的影响,从而导致奖励分配不公平的问题。针对此问题提出了一种基于隐式通信的值分解多智能体强化学习方法(Value Function Factorization for Multi-Agent Reinforcement Learning based on Implicit Communication,VFRL-IC),通过挖掘智能体之间的局部关系,缓解环境不确定性问题带来的影响:首先,提出隐式通信框架,在训练阶段使智能体共享局部观测信息以训练局部策略;其次,基于局部观测信息构建全局影响的评估模型,求解智能体间影响值;最后,设计了一种类多头注意力机制的网络结构,融合智能体间影响值,求解包含全局信息的局部动作值模型。在星际争霸环境中进行实验验证,结果表明,VFRL-IC在各场景中的平均成功率优于基线算法15%∼40%,效率提高18%。

关键词: 值分解, 多智能体强化学习, 部分可观测性, 不确定性, 隐式通信

Abstract: In multi-agent systems, agents typically can only observe partial state information, resulting in each agent making decisions without a complete understanding of the behavior of the other agents and the dynamics of the environment, which in turn increases the difficulty of collaboration. Although the multi-agent reinforcement learning method based on value function factorization has some advantages in solving the local observability problem, there still exists the effect of collaborative uncertainty in multi-agent systems due to high-dimensional state-action space, the complexity of model structure, and so on, which leads to unfair reward assignment. This paper proposes a Value function Factorization multi-agent Reinforcement Learning method based on Implicit Communication (VFRL-IC) to address the uncertainty problem in multi-agent systems by exploring the local relationships between agents. Firstly, an implicit communication framework is proposed to share the local observation information of agents to others during training in order to train the local policy. Secondly, an assessment model of global influence based on local observations of all agents is constructed to solve for inter-agent influence values. Finally, a multi-head attention-like mechanisms-based network structure is designed to solve the local action value model containing global information by fuse inter-agent influence values. Extensive experiments were conducted in StarCraft II to validate the proposed method. The results show that VFRL-IC outperforms baselines by 15% to 40% in success rate across various scenarios, with a 18% increase in efficiency.

Key words: Value Function Factorization, multi-agent reinforcement learning, partial observability, uncertainty, implicit communication