动态一致自信的深度半监督学习

doi:10.3778/j.issn.1673-9418.2104121

计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (11): 2557-2564.DOI: 10.3778/j.issn.1673-9418.2104121

动态一致自信的深度半监督学习

李勇¹^,², 高灿¹^,²^,⁺(), 刘子荣¹^,², 罗金涛¹^,²

1.深圳大学计算机与软件学院，广东深圳 518060
2.广东省智能信息处理重点实验室，广东深圳 518060

收稿日期:2021-05-08 修回日期:2021-06-25 出版日期:2022-11-01 发布日期:2021-05-31
通讯作者: + E-mail: 2005gaocan@163.com
作者简介:李勇（1996—），男，河南周口人，硕士研究生，主要研究方向为半监督学习、计算机视觉。
高灿（1983—），男，湖南南县人，博士，副研究员，硕士生导师，主要研究方向为机器学习、计算机视觉。
刘子荣（1997—），男，广东东莞人，硕士研究生，主要研究方向为机器学习、计算机视觉。
罗金涛（1997—），男，广东梅州人，硕士研究生，主要研究方向为半监督学习、计算机视觉。
基金资助:
国家自然科学基金(61806127);国家自然科学基金(62076164);佛山市教育局项目(2019XJZZ05)

Dynamically Consistent and Confident Deep Semi-supervised Learning

LI Yong¹^,², GAO Can¹^,²^,⁺(), LIU Zirong¹^,², LUO Jintao¹^,²

1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, Guangdong 518060, China
2. Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen, Guangdong 518060, China

Received:2021-05-08 Revised:2021-06-25 Online:2022-11-01 Published:2021-05-31
About author:LI Yong, born in 1996, M.S. candidate. His research interests include semi-supervised learning and computer vision.
GAO Can, born in 1983, Ph.D., associate researcher, M.S. supervisor. His research interests include machine learning and computer vision.
LIU Zirong, born in 1997, M.S. candidate. His research interests include machine learning and computer vision.
LUO Jintao, born in 1997, M.S. candidate. His research interests include semi-supervised learning and computer vision.
Supported by:
National Natural Science Foundation of China(61806127);National Natural Science Foundation of China(62076164);Project of Bureau of Education of Foshan(2019XJZZ05)

摘要/Abstract

摘要：

基于一致性正则化和熵最小化的深度半监督学习方法可以有效提升大型神经网络的性能，减少对标记数据的需求。然而，现有一致性正则化方法的正则损失没有考虑样本之间的差异及错误预测的负面影响，而熵最小化方法则不能灵活调节预测概率分布。首先，针对样本之间的差别以及错误预测带来的负面影响，提出了新的一致性损失函数，称为动态加权一致性正则化（DWCR），可以实现对无标记数据一致性损失的动态加权。其次，为了进一步调节预测概率分布，提出了新的促进低熵预测的损失函数，称为自信促进损失（SCPL），能灵活调节促进模型输出低熵预测的强度，实现类间的低密度分离，提升模型的分类性能。最后，结合动态加权一致性正则化、自信促进损失与有监督损失，提出了名为动态一致自信（DCC）的深度半监督学习方法。多个数据集上的实验表明，所提出方法的分类性能优于目前较先进的深度半监督学习算法。

关键词: 深度半监督学习, 图像分类, 动态加权一致性, 自信预测, 低密度分离

Abstract:

Deep semi-supervised learning methods based on consistency regularization and entropy minimization can effectively improve the performance of large-scale neural networks and reduce the need for labeled data. However, the regularization losses of the existing consistency regularization methods do not consider the differences between samples and the negative impact of mislabeled predictions, whereas the entropy minimization methods cannot flexibly adjust the prediction probability distribution. Firstly, to alleviate the negative impact of discrepancies between unlabeled samples and mislabeled predictions, a new consistency loss function named dynamically weighted consistency regularization (DWCR) is proposed, which can dynamically weigh the consistency loss of unlabeled samples. Then, to further adjust the prediction probability distribution, a new loss function called self-confidence promotion loss (SCPL) is proposed, which can flexibly adjust the strength of the model to generate low entropy prediction and achieve low-density separation between classes, thus improving classification performance. Finally, a deep semi-supervised learning method named dynamic consistency and confidence (DCC) is proposed by combining the DWCR, SPCL and supervised loss. Experiments on several datasets show that the proposed method achieves better classification performance than the state-of-the-art deep semi-supervised learning methods.

Key words: deep semi-supervised learning, image classification, dynamic weighted consistency, confident prediction, low density separation

中图分类号:

TP391

李勇, 高灿, 刘子荣, 罗金涛. 动态一致自信的深度半监督学习[J]. 计算机科学与探索, 2022, 16(11): 2557-2564.

LI Yong, GAO Can, LIU Zirong, LUO Jintao. Dynamically Consistent and Confident Deep Semi-supervised Learning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(11): 2557-2564.

图/表 8

图1 DCC模型总体结构

Fig.1 Overall structure of DCC model

图2 CL函数图像

Fig.2 CL function image

图3 信息熵最小化损失和SCPL

Fig.3 Information entropy minimization loss and SCPL

表1 Error rates of different semi-supervised methods on CIFAR10 单位：%

Table 1

方法	标记数目
方法	250	1 000	4 000
$Π$ -model	53.02±2.05	31.53±0.98	17.41±0.37
PseudoLabel	49.98±1.17	30.91±1.73	16.21±0.11
Mixup	47.43±0.92	25.72±0.66	13.15±0.20
VAT	36.03±2.82	18.68±0.40	11.05±0.31
MeanTeacher	47.32±4.71	17.32±4.00	10.36±0.25
MixMatch	11.08±0.87	7.75±0.32	6.24±0.06
EnAET	7.36±0.34	6.95±0.00	5.35±0.00
DCC	5.00±0.10	4.69±0.01	4.15±0.04

表1 Error rates of different semi-supervised methods on CIFAR10 单位：%

Table 1

方法	标记数目
方法	250	1 000	4 000
$Π$ -model	53.02±2.05	31.53±0.98	17.41±0.37
PseudoLabel	49.98±1.17	30.91±1.73	16.21±0.11
Mixup	47.43±0.92	25.72±0.66	13.15±0.20
VAT	36.03±2.82	18.68±0.40	11.05±0.31
MeanTeacher	47.32±4.71	17.32±4.00	10.36±0.25
MixMatch	11.08±0.87	7.75±0.32	6.24±0.06
EnAET	7.36±0.34	6.95±0.00	5.35±0.00
DCC	5.00±0.10	4.69±0.01	4.15±0.04

表2 Error rates of different semi-supervised methods on SVHN 单位：%

Table 2

方法	标记数目
方法	250	1 000	4 000
$Π$ -model	17.65±0.27	8.60±0.18	5.57±0.14
PseudoLabel	21.16±0.88	10.19±0.41	5.71±0.07
Mixup	39.97±1.89	16.79±0.63	7.96±0.14
VAT	8.41±1.01	5.98±0.21	4.20±0.15
MeanTeacher	6.45±2.43	3.75±0.10	3.39±0.11
MixMatch	3.78±0.26	3.27±0.31	2.89±0.06
EnAET	3.21±0.21	2.92±0.00	2.69±0.00
DCC	2.91±0.05	2.72±0.02	2.62±0.02

表2 Error rates of different semi-supervised methods on SVHN 单位：%

Table 2

方法	标记数目
方法	250	1 000	4 000
$Π$ -model	17.65±0.27	8.60±0.18	5.57±0.14
PseudoLabel	21.16±0.88	10.19±0.41	5.71±0.07
Mixup	39.97±1.89	16.79±0.63	7.96±0.14
VAT	8.41±1.01	5.98±0.21	4.20±0.15
MeanTeacher	6.45±2.43	3.75±0.10	3.39±0.11
MixMatch	3.78±0.26	3.27±0.31	2.89±0.06
EnAET	3.21±0.21	2.92±0.00	2.69±0.00
DCC	2.91±0.05	2.72±0.02	2.62±0.02

表3 Error rates of different semi-supervised methods on CIFAR100 单位：%

Table 3

方法	标记数目
方法	1 000	5 000	10 000
$Π$ -model	—	—	39.19±0.36
Temporal Ensembling	—	—	38.65±0.51
EnAET	58.73±0.00	31.83±0.00	26.93±0.21
DCC	48.66±0.70	31.79±0.21	28.21±0.05

表3 Error rates of different semi-supervised methods on CIFAR100 单位：%

Table 3

方法	标记数目
方法	1 000	5 000	10 000
$Π$ -model	—	—	39.19±0.36
Temporal Ensembling	—	—	38.65±0.51
EnAET	58.73±0.00	31.83±0.00	26.93±0.21
DCC	48.66±0.70	31.79±0.21	28.21±0.05

图4 消融实验结果

Fig.4 Ablation experiment results

图5 DCC对参数的敏感性

Fig.5 Sensitivity of DCC to parameters

参考文献 16

[1]	SOHN K, BERTHELOT D, LI C L, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[C]// Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Dec 6-12, 2020. Red Hook: Curran Associates, 2020: 596-608.
[2]	LEE D. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks[C]// Proceedings of the 30th International Conference on Machine Learning, Atlanta, Jun 16-21, 2013.
[3]	王曙燕, 金航, 孙家泽. GAN图像对抗样本生成方法[J]. 计算机科学与探索, 2021, 15(4): 702-711. DOI
	WANG S Y, JIN H, SUN J Z. GAN image adversarial sample generation method[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(4): 702-711.
[4]	RASMUS A, VALPOLA H, HONKALA M, et al. Semi-supervised learning with ladder networks[C]// Proceedings of the 29th International Conference on Neural Information Processing Systems, Montreal, Dec 7-12, 2015. Red Hook: Curran Associates, 2015: 3546-3554.
[5]	WANG X, KIHARA D, LUO J B, et al. EnAET: a self-trained framework for semi-supervised and supervised lear-ning with ensemble transformations[J]. IEEE Transactions on Image Processing, 2021, 30: 1639-1647. DOI URL
[6]	GRANDVALET Y, BENGIO Y. Semi-supervised learning by entropy minimization[C]// Proceedings of the 18th International Conference on Neural Information Processing Systems, Vancouver, Dec 13-18, 2004. Red Hook: Curran Associates, 2004: 529-536.
[7]	REN Z Z, YEH R, SCHWING A. Not all unlabeled data are equal: learning to weight data in semi-supervised learning[C]// Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Dec 6-12, 2020. Red Hook: Curran Associates, 2020: 21786-21797.
[8]	MIYATO T, MAEDA S, KOYAMA M, et al. Virtual adversarial training: a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(8): 1979-1993. DOI PMID
[9]	LAINE S, AILA T. Temporal ensembling for semi-supervised learning[C]// Proceedings of the 5th International Conference on Learning Representations, Toulon, Apr 24-26, 2017.
[10]	TARVAINEN A, VALPOLA H. Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, Dec 4-9, 2017. Red Hook: Curran Associates, 2017: 1195-1204.
[11]	ZHANG H Y, CISSE M, DAUPHIN Y N, et al. Mixup: beyond empirical risk minimization[C]// Proceedings of the 6th International Conference on Learning Representations, Vancouver, Apr 30 - May 3, 2018.
[12]	DEVRIES T, TAYLOR G W. Improved regularization of convolutional neural networks with Cutout[J]. arXiv:1708. 04552, 2017.
[13]	CUBUK E D, ZOPH B, SHLENS J, et al. RandAugment: practical automated data augmentation with a reduced search space[C]// Proceedings of the 30th Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Washington: IEEE Computer Society, 2020: 702-703.
[14]	BERTHELOT D, CARLINI N, GOODFELLOW I, et al. MixMatch: a holistic approach to semi-supervised learning[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Dec 8-14, 2019. Red Hook: Curran Associates, 2019: 5050-5060.
[15]	ZAGORUYKO S, KOMODAKIS N. Wide residual networks[C]// Proceedings of the 27th British Machine Vision Conference, York, Sep 19-22, 2016. Durham: BMVA, 2016.
[16]	SUTSKEVER I, MARTENS J, DAHL G, et al. On the importance of initialization and momentum in deep learning[C]// Proceedings of the 30th International Conference on Machine Learning, Atlanta, Jun 16-21, 2013: 1139-1147.

动态一致自信的深度半监督学习

Dynamically Consistent and Confident Deep Semi-supervised Learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 16

相关文章 15

编辑推荐

Metrics

[1]	安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
[2]	李宽宽, 刘立波. 双线性聚合残差注意力的细粒度图像分类模型[J]. 计算机科学与探索, 2022, 16(4): 938-949.
[3]	王飞龙, 刘萍, 张玲, 李钢. 改进胶囊网络的小样本图像分类算法[J]. 计算机科学与探索, 2022, 16(10): 2387-2394.
[4]	张海涛, 柴思敏. 改进双分支胶囊网络的高光谱图像分类[J]. 计算机科学与探索, 2022, 16(10): 2405-2414.
[5]	张梦倩, 张莉. 粗-细两阶段卷积神经网络算法[J]. 计算机科学与探索, 2021, 15(8): 1501-1510.
[6]	刘靖祎, 史彩娟, 涂冬景, 刘帅. 零样本图像分类综述[J]. 计算机科学与探索, 2021, 15(5): 812-824.
[7]	刘晓龙, 王士同. 面向开放集图像分类的模糊域自适应方法[J]. 计算机科学与探索, 2021, 15(3): 515-523.
[8]	杨章静, 王文博, 黄璞, 张凡龙. 基于潜子空间去噪的子空间学习图像分类方法[J]. 计算机科学与探索, 2021, 15(12): 2374-2389.
[9]	安平，冀中，刘西瑶. 任务感知双原型网络的人物交互少样本识别[J]. 计算机科学与探索, 2021, 15(11): 2184-2192.
[10]	李祥霞, 吉晓慧, 李彬. 细粒度图像分类的深度学习方法[J]. 计算机科学与探索, 2021, 15(10): 1830-1842.
[11]	马翔, 邓赵红, 王士同. 多粒度融合的模糊规则系统图像特征学习[J]. 计算机科学与探索, 2021, 15(1): 173-184.
[12]	杨萌林，张文生. 分类激活图增强的图像分类算法[J]. 计算机科学与探索, 2020, 14(1): 149-158.
[13]	陈德运，付立军，张学松，于梁，陈海龙，李骜. 多种表示的图像分类方法[J]. 计算机科学与探索, 2019, 13(12): 2138-2148.
[14]	焦志成，李洁，王颖，高新波. 浅层模糊K均值图像分类网络[J]. 计算机科学与探索, 2015, 9(8): 1018-1024.
[15]	田昊, 李国辉, 廉蔺, 贾立. 用于遥感图像建筑物目标分类的层次匹配核[J]. 计算机科学与探索, 2011, 5(7): 588-594.