使用多分类器的分布式模型重用技术

doi:10.3778/j.issn.1673-9418.2105040

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (10): 2310-2319.DOI: 10.3778/j.issn.1673-9418.2105040

使用多分类器的分布式模型重用技术

李新春¹^,³, 詹德川²^,³^,⁺()

1.南京大学计算机科学与技术系,南京 210023
2.南京大学人工智能学院,南京 210023
3.南京大学计算机软件新技术国家重点实验室,南京 210023

收稿日期:2021-05-06 修回日期:2021-06-22 出版日期:2022-10-01 发布日期:2021-06-24
通讯作者: + E-mail: zhandc@nju.edu.cn
作者简介:李新春（1997—）,男,江苏徐州人,硕士研究生,主要研究方向为机器学习、数据挖掘。
詹德川（1982—）,男,江苏扬州人,博士,教授,主要研究方向为机器学习、数据挖掘。
基金资助:
国家自然科学基金(61773198);国家自然科学基金(61632004)

Distributed Model Reuse with Multiple Classifiers

LI Xinchun¹^,³, ZHAN Dechuan²^,³^,⁺()

1. Department of Computer Science and Technology, Nanjing University, Nanjing 210023, China
2. School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
3. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China

Received:2021-05-06 Revised:2021-06-22 Online:2022-10-01 Published:2021-06-24
About author:LI Xinchun, born in 1997, M.S. candidate. His research interests include machine learning and data mining.
ZHAN Dechuan, born in 1982, Ph.D., professor. His research interests include machine learning and data mining.
Supported by:
National Natural Science Foundation of China(61773198);National Natural Science Foundation of China(61632004)

摘要/Abstract

摘要：

传统的机器学习经常采用数据中心化的方式进行训练,然而由于实际应用中的传输开销或者隐私保护限制,数据越来越呈现分散化、隔离化的趋势。分布式训练学习技术为分散在信息孤岛上的数据融合提供了一种解决方案。然而,由于分散化数据本身具有天然异质性,本地数据分布经常是非独立同分布的（Non-IID）,这给分布式训练带来了挑战。首先,为了应对单一模型难以适配所有异质客户端的难题,在分布式训练的基础上引入了模型重用技术,提出了分布式模型重用框架（DMR）。然后,通过理论分析指出集成学习可以为异构数据提供有效的解决方案,并在此基础之上提出了使用多分类器的分布式模型重用技术（McDMR）。最后,为了减少实际应用过程中的存储、计算和传输开销,继而提出了两种具体的优化方案：使用多头分类器的分布式模型重用（McDMR-MH）和使用随机分类器采样的分布式模型重用（McDMR-SC）。在多个公开数据集上进行实验,实验结果验证了所提方法的有效性。

关键词: 学件, 模型重用, 多分类器, 分布式学习, 集成, 效率, 隐私保护

Abstract:

Traditional machine learning always takes a data centralized training strategy, while the transmission cost or data privacy protection in many real-world applications results in distributed and isolated data. Distributed learning provides an effective solution for efficient data fusion across isolated islands. However, due to the natural heterogeneity in real-world applications, the distributions of local data are not independently and identically distributed (Non-IID), which poses a huge challenge to distributed learning. First of all, to overcome the problem of data heterogeneity across local clients, this paper introduces model reuse into the procedure of distributed training and proposes a distributed model reuse (DMR) framework. Then, this paper theoretically shows that ensemble learning can provide a universal solution to data heterogeneity, and proposes a technique of multiple classifiers based distributed model reuse (McDMR). Finally, in order to reduce the storage, computation and transmission cost in practical applications, this paper further proposes two specific solutions including multi-head classifier and stochastic classifier based McDMR, which are named as McDMR-MH and McDMR-SC respectively. Experimental results on several public datasets verify the superiorities of the proposed methods.

Key words: learnware, model reuse, multiple classifiers, distributed learning, ensemble, efficiency, privacy protection

中图分类号:

TP391

李新春, 詹德川. 使用多分类器的分布式模型重用技术[J]. 计算机科学与探索, 2022, 16(10): 2310-2319.

LI Xinchun, ZHAN Dechuan. Distributed Model Reuse with Multiple Classifiers[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(10): 2310-2319.

图/表 9

图1 分布式模型重用示意图

Fig.1 Illustration of distributed model reuse

图2 使用多分类器的分布式模型重用示意图

Fig.2 Illustration of multiple classifiers based distributed model reuse

图3 所提算法使用的网络结构示意图

Fig.3 Illustration of networks in proposed methods

图4 C100-NonIID场景下客户端数据类别分布图

Fig.4 Illustration of clients’ class distributions in C100-NonIID

图5 基于Mnist数据集的性能对比

Fig.5 Performance comparison on Mnist

图6 基于Cifar10数据集的性能对比

Fig.6 Performance comparison on Cifar10

图7 基于Cifar100数据集的性能对比

Fig.7 Performance comparison on Cifar100

图8 对 S具体设置的扰动实验

Fig.8 Ablation studies on settings of S

表1 C10-NonIID上算法运行效率比较

Table 1 Running time comparison on C10-NonIID

算法	运行时间/min
FedAvg	326
FedProx	357
FLDA	391
McDMR	493
McDMR-MH	366
McDMR-SC	343

参考文献 31

[1]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]// Advances in Neural Information Processing Systems 25, Lake Tahoe, Dec 3-6, 2012. Red Hook: Curran Associates, 2012: 1106-1114.
[2]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[3]	DEAN J, CORRADO G S, MONGA R, et al. Large scale distributed deep networks[C]// Advances in Neural Information Processing Systems 25, Lake Tahoe, Dec 3-6, 2012. Red Hook: Curran Associates, 2012: 1232-1240.
[4]	朱泓睿, 元国军, 姚成吉, 等. 分布式深度学习训练网络综述[J]. 计算机研究与发展, 2021, 58(1): 98-115.
	ZHU H R, YUAN G J, YAO C J, et al. Survey on network of distributed deep learning training[J]. Journal of Computer Research and Development, 2021, 58(1): 98-115.
[5]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communi-cation-efficient learning of deep networks from decentralized data[C]// Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale,Apr 20-22, 2017. New York: ACM, 2017: 1273-1282.
[6]	LI M, ZHOU L, YANG Z, et al. Parameter server for distri-buted machine learning[C]// Advances in Neural Informa-tion Processing Systems 26, Lake Tahoe, Dec 5-10, 2013. Red Hook: Curran Associates, 2013: 2.
[7]	ABADI M, CHU A, GOODFELLOW I, et al. Deep learning with differential privacy[C]// Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Oct 24-28, 2016. New York: ACM, 2016: 308-318.
[8]	LIU Y, KANG Y, XING C, et al. A secure federated transfer learning framework[J]. IEEE Intelligent Systems, 2020, 35(4): 70-82. DOI URL
[9]	ZHOU Z H. Learnware: on the future of machine learning[J]. Frontiers in Computer Science, 2016, 10(4): 589-590.
[10]	ZHAO Y, LI M, LAI L, et al. Federated learning with non-IID data[J]. arXiv:1806.00582, 2018.
[11]	WU X Z, LIU S, ZHOU Z H. Heterogeneous model reuse via optimizing multiparty multiclass margin[C]// Proceedings of the 36th International Conference on Machine Learning,California, Jun 9-15, 2019. New York: ACM, 2019: 6840-6849.
[12]	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. arXiv:1812.06127, 2018.
[13]	YAO X, HUANG C, SUN L. Two-stream federated learning: reduce the communication costs[C]// Proceedings of the 2018 IEEE Visual Communications and Image Processing, Taichung, China, Dec 9-12, 2018. Piscataway: IEEE, 2018: 1-4.
[14]	ARIVAZHAGAN M G, AGGARWAL V, SINGH A K, et al. Federated learning with personalization layers[J]. arXiv:1912.00818, 2019.
[15]	PETERSON D, KANANI P, MARATHE V J. Private federated learning with domain adaptation[J]. arXiv:1912.06733, 2019.
[16]	KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning[C]// Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020: 5132-5143.
[17]	SMITH V, CHIANG C K, SANJABI M, et al. Federated multi-task learning[J]. arXiv:1705.10467, 2017.
[18]	JIANG Y, KONEČNÝ J, RUSH K, et al. Improving federated learning personalization via model agnostic meta learning[J]. arXiv:1909.12488, 2019.
[19]	ZHANG M L, ZHOU Z H. Exploiting unlabeled data to enhance ensemble diversity[J]. Data Mining and Knowledge Discovery, 2013, 26(1): 98-129. DOI URL
[20]	ZHOU Z H, LI N. Multi-information ensemble diversity[C]// LNCS 5997: Proceedings of the 9th International Workshop on Multiple Classifier Systems, Cairo, Apr 7-9, 2010. Berlin, Heidelberg: Springer, 2010: 134-144.
[21]	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[J]. arXiv:1503.02531, 2015.
[22]	GAO H, LI Y, PLEISS G, et al. Snapshot ensembles: train 1, get M for free[J]. arXiv:1704.00109, 2017.
[23]	LI H, NG J Y H, NATSEV P. EnsembleNet: end-to-end optimization of multi-headed models[J]. arXiv:1905.09979, 2019.
[24]	YANG Y, ZHAN D C, FAN Y, et al. Deep learning for fixed model reuse[C]// Proceedings of the 31st Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Menlo Park: AAAI, 2017: 2831-2837.
[25]	YE H J, ZHAN D C, JIANG Y, et al. Rectify heterogeneous models with semantic mapping[C]// Proceedings of the 35th International Conference on Machine Learning, Stockhol-msmässan, Jul 10-15, 2018. New York: ACM, 2018: 1904-1913.
[26]	赵鹏, 周志华. 基于决策树模型重用的分布变化流数据学习[J]. 中国科学: 信息科学, 2021, 51(1): 1-12.
	ZHAO P, ZHOU Z H. Learning from distribution-changing data streams via decision tree model reuse[J]. SCIENTIA SINICA Informationis, 2021, 51(1): 1-12. DOI URL
[27]	李新春, 詹德川. 一种保持语义关系的词向量复用方法[J]. 中国科学: 信息科学, 2020, 50(6): 813-823.
	LI X C, ZHAN D C. A semantic relation preserved word embedding reuse method[J]. SCIENTIA SINICA Informationis, 2020, 50(6): 813-823. DOI URL
[28]	HAMER J, MOHRI M, SURESH A T. FedBoost: a comm-unication-efficient algorithm for federated learning[C]// Proceedings of the 37th International Conference on Machine Learning. New York: ACM, 2020: 3973-3983.
[29]	YOSINSKI J, CLUNE J, BENGIO Y, et al. How transferable are features in deep neural networks?[J]. arXiv:1411.1792, 2014.
[30]	KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[31]	LI X C, ZHAN D C, YANG J Q, et al. Deep multiple instance selection[J]. Science China Information Sciences, 2021, 64(3): 130102. DOI URL

使用多分类器的分布式模型重用技术

Distributed Model Reuse with Multiple Classifiers

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 31

相关文章 15

编辑推荐

Metrics

[1]	陈洋, 王士同. 多样性正则化极限学习机的集成方法[J]. 计算机科学与探索, 2022, 16(8): 1819-1928.
[2]	张壮, 王士同. 不平衡数据的Takagi-Sugeno-Kang模糊分类集成模型[J]. 计算机科学与探索, 2022, 16(6): 1374-1382.
[3]	田静, 杜云明, 李帅, 刘义. Paillier加密的隐私保护群智感知任务发布算法[J]. 计算机科学与探索, 2022, 16(6): 1327-1333.
[4]	申瑞彩, 翟俊海, 侯璎真. 选择性集成学习多判别器生成对抗网络[J]. 计算机科学与探索, 2022, 16(6): 1429-1438.
[5]	陈共驰, 荣欢, 马廷淮. 面向连贯性强化的无真值依赖文本摘要模型[J]. 计算机科学与探索, 2022, 16(3): 621-636.
[6]	韩刚, 吕英泽, 罗维, 王嘉乾. 重大疫情患者隐私数据保护方案研究[J]. 计算机科学与探索, 2022, 16(2): 359-371.
[7]	荣欢, 马廷淮. 利用收益预测与策略梯度两阶段众包评论集成[J]. 计算机科学与探索, 2021, 15(8): 1476-1489.
[8]	郭子菁, 罗玉川, 蔡志平, 郑腾飞. 医疗健康大数据隐私保护综述[J]. 计算机科学与探索, 2021, 15(3): 389-402.
[9]	高昂, 梁英, 谢小杰, 王梓森, 李锦涛. 支持隐私保护的社交网络信息传播方法[J]. 计算机科学与探索, 2021, 15(2): 233-248.
[10]	薛红艳, 钱雪忠, 周世兵. 超簇加权的集成聚类算法[J]. 计算机科学与探索, 2021, 15(12): 2362-2373.
[11]	尤坊州, 白亮. 关键节点选择的快速图聚类算法[J]. 计算机科学与探索, 2021, 15(10): 1930-1937.
[12]	黄宇翔, 黄栋, 王昌栋, 赖剑煌. 基于集成学习的改进深度嵌入聚类算法[J]. 计算机科学与探索, 2021, 15(10): 1949-1957.
[13]	孙伟, 张羽. 利用流挖掘和图挖掘的内网异常检测方法[J]. 计算机科学与探索, 2020, 14(7): 1154-1163.
[14]	韩明明，孙广路，朱素霞. 自适应概念漂移问题的增量集成分类算法[J]. 计算机科学与探索, 2020, 14(7): 1200-1210.
[15]	陈兴国，徐修颖，陈康扬，杨光. 基于CMAES集成学习方法的地表水质分类[J]. 计算机科学与探索, 2020, 14(3): 426-436.