Abstract:

In the field of reinforcement learning, how to balance the relationship between exploration and exploi-tation is a hard problem. The reinforcement learning method proposed in recent years mainly focuses on how to combine the deep learning technology to improve the generalization ability of the algorithm, but ignores the explo-ration-exploitation dilemma. The traditional reinforcement learning method can effectively solve the exploration problem, but there are certain restrictions: the state space of the Markov decision process must be discrete and limited. In this paper, the Bayesian method is proposed to improve the efficiency of deep reinforcement algorithm. And the main contribution is to extend the method of calculating the posterior distribution of parameters in Bayesian linear regression to nonlinear models such as artificial neural networks. By combining Bootstrapped DQN (deep Q-network) and the computational method proposed in this paper, Bayesian Bootstrapped DQN (BBDQN) is obtained. Finally, the results of the experiments in two environments show that BBDQN is more efficient than DQN and Bootstrapped DQN in the face of deep exploration.

Key words: deep reinforcement learning, exploration and exploitation, Bayes?? theorem

摘要：

在强化学习领域，如何平衡探索与利用之间的关系是一个难题。近几年提出的强化学习方法主要关注如何结合深度学习技术来提高算法的泛化能力，却忽略探索利用困境这一问题。传统的强化学习方法可以有效解决探索问题，但存在着一定的限制条件：马尔可夫决策过程的状态空间必须是离散并有限的。提出通过贝叶斯方法来提高深度强化算法的探索效率，并将贝叶斯线性回归中计算参数后验分布的方法扩展到人工神经网络等非线性模型中，通过结合Bootstrapped DQN和提出的计算方法得到了贝叶斯自举深度Q网络算法（BBDQN）。最后用两个环境下的实验表明了BBDQN在面对深度探索问题时的探索效率要优于DQN以及Bootstrapped DQN。

关键词: 深度强化学习, 探索与利用, 贝叶斯定理