Incomplete Modality Transfer Learning via Latent Low-Rank Constraint

doi:10.3778/j.issn.1673-9418.2103085

Abstract

Abstract:

When insufficient or incomplete multi-modality data are available in training, the corresponding classi-fication learning may lead to poor training performance or even failure. In order to tackle with this problem, the transfer learning algorithm called IMTL (incomplete modality transfer learning via latent low-rank constraint) is proposed in this paper. The proposed algorithm addresses the incomplete modality problem in two ways. Firstly, latent factors are introduced into a low-rank constrained subspace framework so as to mine missing modality infor-mation on the target domain. With the help of an auxiliary yet complete modality dataset, the proposed cross-modality and cross-dataset transfer learning strategy is used to help align data between modalities or datasets. Sec-ondly, a small amount of labeled target data is used to align the supervision information so as to maintain the internal structure of the target data during the transfer learning. Experimental results show that the proposed algorithm outperforms the previous transfer learning algorithms, and significantly improves the classification performance on the adopted incomplete target datasets.

Key words: transfer learning, incomplete modality, latent low-rank constraint

摘要：

当数据是多模态时，如果在训练阶段没有足够或完整的目标数据可参与训练，则可能导致训练效果较差甚至失败。为了解决该问题，提出了一个基于潜在的低秩约束的不完整模态迁移学习算法（IMTL）。所提算法通过两方面来解决不完整模态问题：一方面，基于低秩约束子空间框架，引入潜在因素来挖掘目标域中缺失的模态信息，然后借助具有完整模态的辅助数据集，通过跨模态或跨数据集方向的迁移学习来帮助模态或数据集之间的数据对齐；另一方面，利用少量标记目标数据来完成监督信息对齐从而保持目标数据在迁移学习过程中的内在结构。实验结果表明，所提算法较之于传统的迁移学习算法有明显优势；即使对于不完整的目标数据，也可以显著地提高分类性能。

关键词: 迁移学习, 不完整模态, 潜在的低秩约束

CLC Number:

TP181

XU Guangsheng, WANG Shitong. Incomplete Modality Transfer Learning via Latent Low-Rank Constraint[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(12): 2775-2787.

徐光生, 王士同. 基于潜在的低秩约束的不完整模态迁移学习[J]. 计算机科学与探索, 2022, 16(12): 2775-2787.

Figures/Tables 6

References 37

[1]	PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2009, 22(10): 1345-1359. DOI URL
[2]	ZHOU Z H. Learning with unlabeled data and its applic-ation to image retrieval[C]// Proceedings of the 2006 Pacific Rim International Conference on Artificial Intelligence. Berlin, Heidelberg: Springer, 2006: 5-10.
[3]	CHEPLYGINA V, DE BRUIJNE M, PLUIM J P W. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis[J]. Medical Image Analysis, 2019, 54: 280-296. DOI PMID
[4]	DAI W, XUE G R, QIANG Y, et al. Co-clustering based classification for out-of-domain documents[C]// Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Jose, Aug 12-15, 2007. New York: ACM, 2007: 210-219.
[5]	程旸, 蒋亦樟, 钱鹏江, 等. 知识迁移的极大熵聚类算法及其在纹理图像分割中的应用[J]. 智能系统学报, 2017, 12(2): 179-187.
	CHENG Y, JIANG Y Z, QIAN P J, et al. A maximum entropy clustering algorithm based on knowledge transfer and its application to texture image segmentation[J]. CAAI Transactions on Intelligent Systems, 2017, 12(2): 179-187.
[6]	ZHUANG F, LUO P, XIONG H, et al. Exploiting asso-ciations between word clusters and document classes for cross-domain text categorization[J]. Statistical Analysis and Data Mining, 2011, 4(1): 100-114. DOI URL
[7]	BLITZER J, MCDONALD R T, PEREIRA F. Domain adaptation with structural correspondence learning[C]// Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Jul 22-23, 2006. Stroudsburg: ACL, 2006: 120-128.
[8]	PAN S J, KWOK J T, YANG Q. Transfer learning via dimensionality reduction[C]// Proceedings of the 23rd AAAI Conference on Artificial Intelligence, Chicago, Jul 13-17, 2008. Menlo Park: AAAI, 2008: 677-682.
[9]	SI S, TAO D, GENG B. Bregman divergence-based regul-arization for transfer subspace learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2009, 22(7): 929-942. DOI URL
[10]	毛发贵, 李碧雯, 沈备军. 基于实例迁移的跨项目软件缺陷预测[J]. 计算机科学与探索, 2016, 10(1): 43-55. DOI URL
	MAO F G, LI B W, SHEN B J. Cross-project software defect prediction based on instance transfer[J]. Journal of Frontiers of Computer Science and Technology, 2016, 10(1): 43-55. DOI URL
[11]	DAI W Y, YANG Q, XUE G R, et al. Boosting for transfer learning[C]// Proceedings of the 24th International Conference on Machine Learning, Corvallis, Jun 20-24, 2007. New York: ACM, 2007: 193-200.
[12]	LIU G, LIN Z, YAN S, et al. Robust recovery of subspace structures by low-rank representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(1): 171-184. DOI URL
[13]	RAAB C, SCHLEIF F M. Domain adaptation via low-rank basis approximation[J]. arXiv:1907.01343, 2019.
[14]	SHAO M, KIT D, FU Y. Generalized transfer subspace learning through low-rank constraint[J]. International Journal of Computer Vision, 2014, 109(1/2): 74-93. DOI URL
[15]	LIU G C, YAN S C. Latent low-rank representation for subspace segmentation and feature extraction[C]// Procee-dings of the 2011 IEEE International Conference on Comp-uter Vision, Barcelona, Nov 6-13, 2011. Washington: IEEE Computer Society, 2011: 1615-1622.
[16]	DING Z M, SHAO M, FU Y. Latent low-rank transfer subspace learning for missing modality recognition[C]// Proceedings of the 28th AAAI Conference on Artificial Intelligence, Québec City, Jul 27-31, 2014. Menlo Park: AAAI, 2014: 1192-1198.
[17]	THOPALLI K, ANIRUDH R, THIAGARAJAN J J, et al. Multiple subspace alignment improves domain adaptation[C]// Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, May 12-17, 2019. Piscataway: IEEE, 2019: 3552-3556.
[18]	BELKIN M, NIYOGI P, SINDHWANI V. Manifold regula-rization: a geometric framework for learning from labeled and unlabeled examples[J]. Journal of Machine Learning Research, 2006, 7: 2399-2434.
[19]	LONG M, WANG J, DING G, et al. Adaptation regulariz-ation: a general framework for transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2013, 26(5): 1076-1089. DOI URL
[20]	YU C N J, JOACHIMS T. Learning structural SVMS with latent variables[C]// Proceedings of the 26th Annual Intern-ational Conference on Machine Learning, Montreal, Jun 14-18, 2009. New York: ACM, 2009: 1169-1176.
[21]	LI J, NAJMI A, GRAY R M. Image classification by a two-dimensional hidden Markov model[J]. IEEE Transactions on Signal Processing, 2000, 48(2): 517-533. DOI URL
[22]	DING Z, SHAO M, FU Y. Missing modality transfer learning via latent low-rank constraint[J]. IEEE Transactions on Image Processing, 2015, 24(11): 4322-4334. DOI PMID
[23]	LIU G C, LIN Z C, YU Y. Robust subspace segmentation by low-rank representation[C]// Proceedings of the 27th Int-ernational Conference on Machine Learning, Haifa, Jun 21-24, 2010. Madison: Omnipress, 2010: 663-670.
[24]	RAZZAGHI P, RAZZAGHI P, ABBASI K. Transfer sub-space learning via low-rank and discriminative reconstruction matrix[J]. Knowledge-Based Systems, 2019, 163: 174-185. DOI URL
[25]	DING Z, SHAO M, FU Y. Incomplete multisource transfer learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 29(2): 310-323. DOI URL
[26]	杨昌健, 邓赵红, 蒋亦樟, 等. 基于迁移学习的癫痫EEG信号自适应识别[J]. 计算机科学与探索, 2014, 8(3): 329-337. DOI URL
	YANG C J, DENG Z H, JIANG Y Z, et al. Adaptive recog-nition of epileptic EEG signals based on transfer learning[J]. Journal of Frontiers of Computer Science and Technology, 2014, 8(3): 329-337.
[27]	PAN S J, TSANG I W, KWOK J T, et al. Domain adaptation via transfer component analysis[J]. IEEE Transa-ctions on Neural Networks, 2010, 22(2): 199-210.
[28]	DING Z, FU Y. Robust multiview data analysis through collective low-rank subspace[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 29(5): 1986-1997. DOI URL
[29]	CAI J F, CANDÈS E J, SHEN Z. A singular value thresh-olding algorithm for matrix completion[J]. SIAM Journal on Optimization, 2010, 20(4): 1956-1982. DOI URL
[30]	SHAO M, FU Y. Hierarchical hyperlingual-words for multi-modality face classification[C]// Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, Shanghai, Apr 22-26, 2013. Washington: IEEE Computer Society, 2013: 1-6.
[31]	JHUO I H, LIU D, LEE D T, et al. Robust visual domain adaptation with low-rank reconstruction[C]// Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 2168-2175.
[32]	GONG B Q, SHI Y, SHA F, et al. Geodesic flow kernel for unsupervised domain adaptation[C]// Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 2066-2073.
[33]	FERNANDO B, HABRARD A, SEBBAN M, et al. Unsup-ervised visual domain adaptation using subspace alignment[C]// Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Dec 1-8, 2013. Washington: IEEE Computer Society, 2013: 2960-2967.
[34]	WANG J D, FENG W J, CHEN Y Q, et al. Visual domain adaptation with manifold embedded distribution alignment[C]// Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Oct 22-26, 2018. New York: ACM, 2018: 402-410.
[35]	TAHMORESNEZHAD J, HASHEMI S. Visual domain adaptation via transfer feature learning[J]. Knowledge and Information Systems, 2017, 50(2): 585-605. DOI URL
[36]	BELHUMEUR P N, HESPANHA J P, KRIEGMAN D J. Eigenfaces vs. Fisherfaces: recognition using class specific linear projection[C]// LNCS 1064: Proceedings of the 4th European Conference on Computer Vision, Cambridge, Apr 15-18, 1996. Berlin, Heidelberg: Springer, 1966: 43-58.
[37]	HE X F, NIYOGI P. Locality preserving projections[C]// Advances in Neural Information Processing Systems 16, Vancouver, Dec 8-13, 2003. Cambridge: MIT Press, 2003: 153-160.

Datasets	Subspace methods	DASA	GFK	RDALR	MEDA	MMTL	IMTL
BUAANIR	PCA	76.15±0.55	74.33±0.65	72.86±0.86	46.33±0.33	75.90±0.38	80.72±0.77
	LPP	71.56±0.22	70.86±0.84	68.73±0.84	71.84±0.26	75.72±0.25	78.68±0.02
	LDA	68.64±0.27	67.41±0.89	60.87±0.05	21.61±0.65	74.64±0.37	77.57±0.21
BUAAVIS	PCA	57.73±0.47	57.21±0.88	59.29±0.90	34.41±0.96	56.20±0.28	66.22±0.34
	LPP	54.81±0.93	62.09±0.56	53.76±0.55	48.09±0.05	55.95±0.98	70.25±0.42
	LDA	71.28±0.29	57.18±0.65	45.72±0.54	22.25±0.14	55.33±0.24	68.92±0.03
OuluNIR	PCA	69.12±0.44	72.20±0.23	71.86±0.09	76.43±0.04	72.91±0.27	78.04±0.73
	LPP	72.30±0.92	75.44±0.08	62.70±0.50	78.63±0.05	71.18±0.63	83.04±0.84
	LDA	34.12±0.81	49.63±0.92	43.10±0.60	68.81±0.02	71.89±0.36	79.39±0.32
OuluVIS	PCA	87.33±0.95	86.99±0.11	86.49±0.14	80.78±0.73	87.37±0.39	92.43±0.75
	LPP	86.39±0.87	84.33±0.04	57.26±0.42	86.07±0.02	88.01±0.79	90.47±0.26
	LDA	71.18±0.09	80.71±0.48	49.60±0.19	87.77±0.01	88.14±0.52	92.87±0.46

Datasets	Subspace methods	DASA	GFK	RDALR	MEDA	MMTL	IMTL
BUAANIR	PCA	76.15±0.55	74.33±0.65	72.86±0.86	46.33±0.33	75.90±0.38	80.72±0.77
	LPP	71.56±0.22	70.86±0.84	68.73±0.84	71.84±0.26	75.72±0.25	78.68±0.02
	LDA	68.64±0.27	67.41±0.89	60.87±0.05	21.61±0.65	74.64±0.37	77.57±0.21
BUAAVIS	PCA	57.73±0.47	57.21±0.88	59.29±0.90	34.41±0.96	56.20±0.28	66.22±0.34
	LPP	54.81±0.93	62.09±0.56	53.76±0.55	48.09±0.05	55.95±0.98	70.25±0.42
	LDA	71.28±0.29	57.18±0.65	45.72±0.54	22.25±0.14	55.33±0.24	68.92±0.03
OuluNIR	PCA	69.12±0.44	72.20±0.23	71.86±0.09	76.43±0.04	72.91±0.27	78.04±0.73
	LPP	72.30±0.92	75.44±0.08	62.70±0.50	78.63±0.05	71.18±0.63	83.04±0.84
	LDA	34.12±0.81	49.63±0.92	43.10±0.60	68.81±0.02	71.89±0.36	79.39±0.32
OuluVIS	PCA	87.33±0.95	86.99±0.11	86.49±0.14	80.78±0.73	87.37±0.39	92.43±0.75
	LPP	86.39±0.87	84.33±0.04	57.26±0.42	86.07±0.02	88.01±0.79	90.47±0.26
	LDA	71.18±0.09	80.71±0.48	49.60±0.19	87.77±0.01	88.14±0.52	92.87±0.46

Datasets	Subspace methods	DASA	GFK	RDALR	MEDA	MMTL	IMTL
CMUHR	PCA	18.81±0.55	23.14±0.31	17.01±0.19	48.64±0.31	39.58±0.67	56.51±0.33
	LPP	27.30±0.63	24.60±0.76	18.71±0.06	51.30±0.10	53.89±0.44	60.16±0.47
	LDA	19.01±0.64	21.73±0.39	20.39±0.19	59.40±0.05	49.82±0.41	57.32±0.50
CMULR	PCA	9.69±0.77	11.30±0.73	10.32±0.83	31.36±0.09	17.08±0.18	37.53±0.25
	LPP	11.21±0.50	12.04±0.03	9.39±0.74	29.41±0.41	33.27±0.40	35.31±0.45
	LDA	16.55±0.09	13.81±0.16	16.38±1.02	25.84±0.04	35.40±0.32	38.36±0.80
YaleBHR	PCA	13.77±0.04	12.61±0.55	13.38±0.55	35.63±0.06	27.14±0.42	41.58±0.55
	LPP	17.95±0.15	18.54±0.80	11.65±0.62	35.51±0.19	36.43±0.40	42.73±0.25
	LDA	11.66±0.52	15.03±0.61	15.25±0.24	32.91±0.07	34.26±0.32	39.35±0.73
YaleBLR	PCA	12.52±0.47	10.94±0.18	11.98±0.21	22.92±0.12	14.22±0.22	28.54±0.23
	LPP	11.54±0.80	11.54±0.52	12.94±0.06	28.24±0.58	24.67±0.36	30.69±0.99
	LDA	10.81±0.30	12.15±0.81	17.82±0.97	23.87±0.28	25.88±0.05	29.95±0.41

Datasets	Subspace methods	DASA	GFK	RDALR	MEDA	MMTL	IMTL
CMUHR	PCA	18.81±0.55	23.14±0.31	17.01±0.19	48.64±0.31	39.58±0.67	56.51±0.33
	LPP	27.30±0.63	24.60±0.76	18.71±0.06	51.30±0.10	53.89±0.44	60.16±0.47
	LDA	19.01±0.64	21.73±0.39	20.39±0.19	59.40±0.05	49.82±0.41	57.32±0.50
CMULR	PCA	9.69±0.77	11.30±0.73	10.32±0.83	31.36±0.09	17.08±0.18	37.53±0.25
	LPP	11.21±0.50	12.04±0.03	9.39±0.74	29.41±0.41	33.27±0.40	35.31±0.45
	LDA	16.55±0.09	13.81±0.16	16.38±1.02	25.84±0.04	35.40±0.32	38.36±0.80
YaleBHR	PCA	13.77±0.04	12.61±0.55	13.38±0.55	35.63±0.06	27.14±0.42	41.58±0.55
	LPP	17.95±0.15	18.54±0.80	11.65±0.62	35.51±0.19	36.43±0.40	42.73±0.25
	LDA	11.66±0.52	15.03±0.61	15.25±0.24	32.91±0.07	34.26±0.32	39.35±0.73
YaleBLR	PCA	12.52±0.47	10.94±0.18	11.98±0.21	22.92±0.12	14.22±0.22	28.54±0.23
	LPP	11.54±0.80	11.54±0.52	12.94±0.06	28.24±0.58	24.67±0.36	30.69±0.99
	LDA	10.81±0.30	12.15±0.81	17.82±0.97	23.87±0.28	25.88±0.05	29.95±0.41