Abstract:

Reinforcement learning (RL) has achieved great success on sequential decision-making problems. Along with the fast advances of RL, transfer learning (TL) arises as an important technique to accelerate the learning process of RL by leveraging and transferring external knowledge. Policy transfer is a kind of transfer learning approach, in which the external knowledge is teacher policies from source tasks. Existing policy transfer methods either transfer knowledge by measuring similarities between source and target tasks, or select best policy by estimating performance of source policies on target task. However, performance estimation can sometimes be unreliable, which can lead to negative transfer. To solve this problem, this paper develops a novel policy transfer method called stochastic ensemble policy transfer (SEPT), which generates a teacher policy to make transfer instead of choosing a policy from source policy library. SEPT changes policy transfer into the option learning problem to get termination probability. Then teacher policy is integrated from policy library and probability weights of source policies are calculated from termination probability. The knowledge of teacher policy is transferred by policy distillation. Experimental results show SEPT accelerates RL effectively and outperforms other state-of-the-art policy transfer methods in both discrete and continuous action spaces.

Key words: transfer learning (TL), reinforcement learning (RL), policy transfer, option learning, ensemble, policy distillation

摘要：

强化学习（RL）在序列决策问题上取得了巨大的成功。随着强化学习的飞速发展，迁移学习（TL）成为了一种重要的可以通过利用和转移外部知识来加速强化学习的技术。策略迁移是一种外部知识来自教师策略的迁移强化学习方法。现有的策略转移方法要么通过测量源任务与目标任务之间的相似性来转移知识，要么通过估计源策略在目标任务上的性能来选择最佳源策略。但是，性能估计有时可能不可靠，这可能会导致负迁移。针对这种问题，提出了一种新的策略转移方法，称为随机集成策略迁移（SEPT）。SEPT不是在源策略库中选择一个策略，而是利用源策略集成出教师策略来进行迁移。SEPT把策略迁移转变为选项学习问题以便获得终止概率，用终止概率计算出源策略的概率权重，根据概率权重从策略库中集成出教师策略。然后，通过策略蒸馏的方式从教师策略进行知识迁移。实验结果表明SEPT可以有效地加速强化学习训练，并且在离散和连续空间上都能胜过其他最佳的策略迁移方法。

关键词: 迁移学习（TL）, 强化学习（RL）, 策略迁移, 选项学习, 集成, 策略蒸馏