Transfer Learning Boosting for Weight Optimization Under Multi-source Domain Distribution

doi:10.3778/j.issn.1673-9418.2112032

Abstract

Abstract: The deep decision tree transfer learning Boosting method (DTrBoost) can only adapt to the training data of one source domain and one target domain, and can not adapt to the samples of several different distribution source domains. In addition, the DTrBoost method synchronously learns data from the source domain to the target domain model, and does not quantify the weight of the learned knowledge according to the degree of importance. In practice, the distribution of data divided according to one or some characteristics of a certain dataset is often inconsistent, the importance of these different distributions to the final model is also inconsistent, and the weight of knowledge transfer is therefore not equal. To solve this problem, a transfer learning method of multi-source domain weight optimization is proposed. The main idea is to calculate the KL divergence distance to the target domain according to the source domain space of different distributions, and calculate the learning weight proportion parameters of the source domain samples of different distributions by using the ratio of KL divergence, so as to optimize the overall gradient function and make the learning direction towards the direction of the fastest gradient decline. The gradient descent algorithm can make the model converge quickly, and ensure the learning speed as well as the transfer learning effect. Experimental results show that the algorithm proposed in this paper adaptively achieves better average performance on the whole. The average classification error rate on all the adopted datasets decreases by 0.013 and even 0.030 on OCR dataset.

Key words: deep decision tree transfer learning Boosting method (DTrBoost), multi-source domain transfer learning, KL divergence, decision tree

摘要： 深度决策树迁移学习Boosting方法（DTrBoost）仅能适应一个源域与一个目标域的训练数据，无法适应多个不同分布的源域的样本。此外，DTrBoost方法同步地从源域中学习数据至目标域模型，并没有根据重要程度量化学习知识的权重。在实践中，对于某数据集的数据按照某一或某些特征划分出来的数据往往分布不一致，并且这些不同分布的数据对于最终模型的重要性也不一致，知识迁移的权重也因此不平等。针对这一问题，提出了多源域优化权重的迁移学习方法，主要思想是根据不同分布的源域空间计算出到目标域的KL距离，利用KL距离的比值计算出不同分布的源域样本的学习权重比例，从而优化整体梯度函数，使学习方向朝着梯度下降最快的方向进行。使用梯度下降算法能使模型较快收敛，在确保迁移学习效果的同时，也能确保学习的速度。实验结果表明，提出的算法在整体上实现了更好的性能并且对于不同的训练数据能够实现自适应效果，分类错误率平均下降0.013，在效果最好的OCR数据集上下降0.030。

关键词: 深度决策树迁移学习Boosting方法（DTrBoost）, 多源域迁移学习, KL距离, 决策树

LI Yunbo, WANG Shitong. Transfer Learning Boosting for Weight Optimization Under Multi-source Domain Distribution[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(6): 1441-1452.

李赟波, 王士同. 多源域分布下优化权重的迁移学习Boosting方法[J]. 计算机科学与探索, 2023, 17(6): 1441-1452.

References

[1] ZHUANG F, QI Z, DUAN K, et al. A comprehensive survey on transfer learning[J]. Proceedings of the IEEE, 2021, 109(1): 43-76.
[2] ZHAO S, CAO Q, CHEN J, et al. A multi-ATL method for transfer learning across multiple domains with arbitrarily different distribution[J].Knowledge-Based Systems, 2016, 94: 60-69.
[3] GUO L, LEI Y, XING S, et al. Deep convolutional transfer learning network: a new method for intelligent fault diagnosis of machines with unlabeled data[J]. IEEE Transactions on Industrial Electronics, 2019, 66(9): 7316-7325.
[4] LIU F, LU J, HAN B, et al. Butterfly: one-step approach towards wildly unsupervised domain adaptation[J]. arXiv:1905.07720, 2019.
[5] WANG Z R, DAI Z H, PóCZOS B, et al. Characterizing and avoiding negative transfer[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 11293- 11302.
[6] SCOTT C. A generalized Neyman-Pearson criterion for optimal domain adaptation[C]//Proceedings of the 30th International Conference on Algorithmic Learning Theory, Chicago, Mar 22-24, 2019: 738-761.
[7] TOKUOKA Y, SUZUKI S, SUGAWARA Y. An inductive transfer learning approach using cycle-consistent adversarial domain adaptation with application to brain tumor segmentation[C]//Proceedings of the 6th International Conference on Biomedical and Bioinformatics Engineering, Shanghai, Nov 13-15, 2019. New York: ACM, 2019: 44-48.
[8] BUCCI S, D’INNOCENTE A, LIAO Y J, et al. Self-supervised learning across domains[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 5516-5528.
[9] MORENO-MU?OZ P, ARTéS-RODRíGUEZ A, áLVAREZ M A. Modular Gaussian processes for transfer learning[C]//Proceedings of the Annual Conference on Neural Information Processing Systems 2021, Dec 6-14, 2021: 24730-24740.
[10] FRIEDMAN J H. Greedy function approximation: a gradient Boosting machine[J]. Annals of Statistics, 2001, 29(5): 1189-1232.
[11] ZHANG X, ZHUANG Y, WANG W, et al. Transfer Boosting with synthetic instances for class imbalanced object recognition[J]. IEEE Transactions on Cybernetics, 2016, 48(1): 357-370.
[12] FREUND Y, SCHAPIRE R E. A decision-theoretic generalization of on-line learning and an application to Boosting[J]. Journal of Computer and System Sciences, 1997, 55(1): 119-139.
[13] CORTES C, MOHRI M, SYED U. Deep Boosting[C]// Proceedings of the 31st International Conference on Machine Learning, Beijing, Jun 21-26, 2014: 1179-1187.
[14] DAI W Y, YANG Q, XUE G R, et al. Boosting for transfer learning[C]//Proceedings of the 24th International Confer-ence on Machine Learning, Corvallis, Jun 20-24, 2007. New York: ACM, 2007: 193-200.
[15] JIANG S, MAO H, DING Z, et al. Deep decision tree transfer Boosting[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(2): 383-395.
[16] 周晶雨, 王士同. 对不平衡数据的多源在线迁移学习算法[J]. 计算机科学与探索, 2023, 17(3): 687-700.
ZHOU J Y, WANG S T. Multi-source online transfer learning algorithm for imbalanced data[J]. Journal of Frontiers of Com- puter Science and Technology, 2023, 17(3): 687-700.
[17] 徐光生, 王士同. 基于潜在的低秩约束的不完整模态迁移学习[J]. 计算机科学与探索, 2022, 16(12): 2775-2787.
XU G S, WANG S T. Incomplete modality transfer learning via latent low-rank constraint[J]. Journal of Frontiers of Com-puter Science and Technology, 2022, 16(12): 2775-2787.
[18] 林佳伟, 王士同. 用于无监督域适应的深度对抗重构分类网络[J]. 计算机科学与探索, 2022, 16(5): 1107-1116.
LIN J W, WANG S T. Deep adversarial-reconstruction-classification networks for unsupervised domain adaptation[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(5): 1107-1116.
[19] KULLBACK S, LEIBLER R A. On information and suf-ficiency[J]. Annals of Mathematical Statistics, 1951, 22(1): 79-86.
[20] BARTLETT P L, MENDELSON S. Rademacher and Gaussian complexities: risk bounds and structural results[J]. Journal of Machine Learning Research, 2002, 3: 463-482.
[21] BAYRAKSAN G, LOVE D K. Data-driven stochastic programming using Phi-divergences[M]//The Operations Research Revolution: Tutorials in Operations Research, 2015: 1-19.
[22] WANG J D, CHEN Y Q, YU H, et al. Easy transfer learning by exploiting intra-domain structures[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo, Shanghai, Jul 8-12, 2019. Piscataway: IEEE, 2019: 1210-1215.
[23] AL-STOUHI S, REDDY C K. Adaptive Boosting for transfer learning using dynamic updates[C]//LNCS 6911: Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Athens, Sep 4-8, 2011. Berlin, Heidelberg: Springer, 2011: 60-75.