Application Analysis of Ensemble Deep Reinforcement Learning in Portfolio Optimization of Stock Index

doi:10.3778/j.issn.1673-9418.2403060

Abstract

Abstract: Portfolio selection based on ensemble deep reinforcement learning is one of the key technologies in the current field of quantitative finance. However, the existing ensemble rolling window method, which determines the next-stage agent based on the optimal indicators from the previous window, has certain lags. To effectively address this issue, a two-layer nested ensemble reinforcement learning method is proposed. This method adopts a two-layer nested pattern for three agents (actor-critic, deep deterministic policy gradient, and proximal policy optimization). The first layer of integration selects stage models through optimal Sharpe ratio optimization, while the second layer integrates the three deep reinforcement learning algorithms through weighted voting. Multiple model snapshots are collected from a single training, and these models are used for integrated prediction during training. This paper conducts a portfolio study on the Shanghai Stock Exchange 50 Investment Index and the Dow Jones Industrial Average Index along with the stocks they contain, respectively, taking the passive strategy of holding the index and the mean-variance portfolio strategy as baseline strategies. Metrics such as portfolio value, annualized return rate, annualized volatility, maximum drawdown, and Sharpe ratio are used as comparative indicators. The results show that the proposed ensemble method performs well in terms of practicality and effectiveness.

Key words: stock portfolio, trading strategy, deep reinforcement learning, two-layer nested ensemble deep reinforcement learning methods, ensemble learning

摘要： 基于集成深度强化学习的投资组合选择是当前量化金融领域的关键技术之一。然而，目前采用上一窗口阶段最优指标决定下一阶段代理的集成滚动窗口方法存在一定的滞后性。为了有效应对这一不足，提出了双层嵌套集成深度强化学习方法。该方法对三种代理（优势演员-评论员、深度确定性策略梯度和近端策略优化）进行两层嵌套模式，第一层集成通过最优化夏普比率进行阶段模型选择，第二层通过加权投票的方法集成三种深度强化学习算法，从单次训练中收集多个模型快照，在训练期间利用这些模型进行集成预测。分别对上证50投资指数和道琼斯指数及其包含的股票进行了投资组合研究，将持有指数被动策略和均值方差投资组合策略作为基线策略。实验采用了投资组合价值、年化回报率、年化波动率、最大回撤和夏普比率等指标作为对比指标。结果表明，所提出的集成方法在实用性和有效性上表现出较好的性能。

关键词: 股票投资组合, 交易策略, 深度强化学习, 双层嵌套集成深度强化学习方法, 集成学习

JI Zhong, ZHANG Wenjia. Application Analysis of Ensemble Deep Reinforcement Learning in Portfolio Optimization of Stock Index[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(1): 237-244.

冀中, 张文嘉. 集成深度强化学习在股票指数投资组合优化中的应用分析[J]. 计算机科学与探索, 2025, 19(1): 237-244.

References

[1] BONNE G, WANG J, ZHANG H. Machine learning factors: capturing nonlinearities in linear factor models[EB/OL]. [2024-01-12]. https://www.msci.com/www/research-report/machine-learning-factors/02410413451.
[2] MENG T L, KHUSHI M. Reinforcement learning in financial markets[J]. Data, 2019, 4(3): 110.
[3] XIONG Z, LIU X Y, ZHONG S, et al. Practical deep reinforcement learning approach for stock trading[EB/OL]. [2024-01-12]. https://arxiv.org/abs/1811.07522.
[4] BJORCK J, GOMES C P, WEINBERGER K Q. Is high variance unavoidable in RL? A case study in continuous control[EB/OL]. [2024-01-12]. https://arxiv.org/abs/2110.11222.
[5] YUN K K, YOON S W, WON D. Prediction of stock price direction using a hybrid GA-XGBoost algorithm with a three-stage feature engineering process[J]. Expert Systems with Applications, 2021, 186: 115716.
[6] MALKIEL B G. The efficient market hypothesis and its critics[J]. Journal of Economic Perspectives, 2003, 17(1): 59-82.
[7] TSAI C F, HSIAO Y C. Combining multiple feature selection methods for stock prediction: union, intersection, and multi-intersection approaches[J]. Decision Support Systems, 2010, 50(1): 258-269.
[8] SUN J, LIU J, LI C, et al. An identification method for vulnerable lines based on combination weighting method and GraphSAGE algorithm[J]. International Journal of Electrical Power & Energy Systems, 2023, 149: 109035.
[9] YANG H, LIU X Y, ZHONG S, et al. Deep reinforcement learning for automated stock trading: an ensemble strategy[C]//Proceedings of the 1st ACM International Conference on AI in Finance. New York: ACM, 2020: 1-8.
[10] LIU X Y, YANG H, GAO J, et al. FinRL: deep reinforcement learning framework to automate trading in quantitative finance[C]//Proceedings of the 2nd ACM International Conference on AI in Finance. New York: ACM, 2021: 1-9.
[11] SUTTON R S, BARTO A G, WILLIAMS R J. Reinforcement learning is direct adaptive optimal control[J]. IEEE Control Systems Magazine, 1992, 12(2): 19-22.
[12] MOODY J, WU L, LIAO Y, et al. Performance functions and reinforcement learning for trading systems and portfolios[J]. Journal of Forecasting, 1998, 17(5/6): 441-470.
[13] MOODY J, SAFFELL M. Learning to trade via direct reinforcement[J]. IEEE Transactions on Neural Networks, 2001, 12(4): 875-889.
[14] DENG Y, BAO F, KONG Y, et al. Deep direct reinforcement learning for financial signal representation and trading[J]. IEEE Transactions on Neural Networks and learning systems, 2016, 28(3): 653-664.
[15] JANG J, SEONG N Y. Deep reinforcement learning for stock portfolio optimization by connecting with modern portfolio theory[J]. Expert Systems with Applications, 2023, 218: 119556.
[16] BISHT K, KUMAR A. Deep reinforcement learning based multi-objective systems for financial trading[C]//Proceedings of the 2020 5th IEEE International Conference on Recent Advances and Innovations in Engineering. Piscataway: IEEE, 2020: 1-6.
[17] WANG Z, HUANG B, TU S, et al. DeepTrader: a deep reinforcement learning approach for risk-return balanced portfolio management with market conditions embedding[C]//Proceedings of the 2021 AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2021: 643-650.
[18] SAPHAL R, RAVINDRAN B, MUDIGERE D, et al. SEERL: sample efficient ensemble reinforcement learning[EB/OL]. [2024-01-12]. https://arxiv.org/abs/2001.05209.
[19] HAO Z, ZHANG H, ZHANG Y. Stock portfolio management by using fuzzy ensemble deep reinforcement learning algorithm[J]. Journal of Risk and Financial Management, 2023, 16(3): 201.
[20] LEEM J B, KIM H Y. Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning[J]. PLoS One, 2020, 15(7): e0236178.
[21] CARTA S, CORRIGA A, FERREIRA A, et al. A multi-layer and multi-ensemble stock trader using deep learning and deep reinforcement learning[J]. Applied Intelligence, 2021, 51: 889-905.
[22] YU X, WU W, LIAO X, et al. Dynamic stock-decision ensemble strategy based on deep reinforcement learning[J]. Applied Intelligence, 2023, 53(2): 2452-2470.