Multi-model Adaptation Method of Possibilistic Clustering Assumption

doi:10.3778/j.issn.1673-9418.2112076

Abstract

Abstract: Graph based semi-supervised learning (GSSL) has been attracting more and more attention with its intui-tiveness and good learning performance in the machine learning community. However, it is found that existing graph based semi-supervised learning method has the problem of poor robustness and sensitivity to noise and abnor-mal data by analysis. In addition, the premise for the GSSL to have good performance is that the training data and test data are independently identically distribution (IID), which leads to some limitations in practical applications. In order to solve above problems, this paper proposes a novel clustering method based on structure risk minimization model, called a multi-model adaptation method of possibilistic clustering assumption (MA-PCA), and effectively minimizes the influence from the noise and abnormal instances based on different data distributions in some reproduced kernel Hilbert space. Its main ideas are as follows: the negative impact of noise and abnormal data on the method is reduced through fuzzy entropy; considering the effective multi-model adaptive learning of training data and test data in the same distribution and different distributions, it can also obtain good performance by rela-xing the constraint of IID between training data and test data; the algorithm implementation and convergence the-orem are given. A large number of experiments and in-depth analysis on multiple real visual datasets show that the proposed method has superior or comparable robustness and generalization performance.

Key words: graph based semi-supervised learning (GSSL), multi-model adaptation, possibilistic clustering, fuzzy entropy

摘要： 基于图的半监督学习（GSSL）凭借其直观性和良好的学习性能，在机器学习领域吸引了越来越多的关注。然而，通过分析发现，现有基于图的半监督学习方法存在对噪声、异常数据的鲁棒性不够好以及较敏感的问题。此外，该方法具有较好性能的前提是训练数据与测试数据为独立同分布（IID），导致在实际应用中存在一定的局限性。为解决上述问题，在某个再生核Hilbert空间，在充分考虑最小化噪声、异常数据影响的基础上，结合不同数据分布特点，基于结构风险最小化模型，提出一种基于可能性聚类假设的多模型适应学习方法（MA-PCA）。其主要思想为：通过模糊熵减弱噪声、异常数据对方法所带来的负面影响；综合考虑训练数据与测试数据在独立同分布和在独立不同分布时进行有效的多模适应学习，弱化训练数据和测试数据的独立同分布约束条件亦具有较好性能；给出了算法实现及其收敛性定理。在多个真实视觉数据集上分别进行了大量实验并进行深入分析，证实了所提方法具有优越的或可比较的鲁棒性和泛化性能。

关键词: 基于图的半监督学习（GSSL）, 多模适应, 可能性聚类, 模糊熵

DAN Yufang, TAO Jianwen, ZHAO Yue, PAN Jie, ZHAO Baoqi. Multi-model Adaptation Method of Possibilistic Clustering Assumption[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(6): 1329-1342.

但雨芳, 陶剑文, 赵悦, 潘婕, 赵宝奇. 可能性聚类假设的多模适应学习方法[J]. 计算机科学与探索, 2023, 17(6): 1329-1342.

References

[1] PAN S J, YANG Q. A survey on transfer learning[J].?IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
[2] ZHU X. Semi-supervised learning literature survey[R]. Madi-son: University of Wisconsin-Madison, 2005.
[3] LIU W, WANG J, CHANG S F. Robust and scalable graph-based semi-supervised learning[J]. Proceedings of the IEEE, 2012, 100(9): 2624-2638.
[4] ZHOU D Y, BOUSQUET O, LAL T N, et al. Learning with local and global consistency[C]//Advances in Neural Infor-mation Processing Systems 16, Vancouver, Dec 8-13, 2003.Cambridge: MIT Press, 2003: 321-328.
[5] MIKHAIL B, PARTHA N, VIKAS S. Manifold regulariza-tion: a geometric framework for learning from labeled and unlabeled examples[J]. The Journal of Machine Learning Res-earch, 2006, 7(1): 2399-2434.
[6] MIKHAIL B, PARTHA N. Laplacian eigenmaps and spectral techniques for embedding and clustering[C]//Advances in Neural Information Processing Systems 14, Vancouver, Dec 3-8, 2001. Cambridge: MIT Press, 2001: 585-591.
[7] TENENBAUM J B, SILVA V D, LANGFORD J C. A global geometric framework for nonlinear dimensionality reduc-tion[J].Science, 2000, 290(500): 2319-2323.
[8] CHAPELLE O, SCHOLKOPF B, ZIEN A. Semi-supervised learning[J]. IEEE Transactions on Neural Networks, 2009, 20(3): 542.
[9] ZHU X, GOLBDERG A. Introduction to semi-supervised learning[M]. San Rafael: Morgan & Claypool Publishers, 2009.
[10] WANG Y Y, CHEN S C, ZHOU Z H. New semi-supervised classification method based on modified cluster assumption[J]. IEEE Transactions on Neural Networks and Learning Systems, 2012, 23(5): 689-702.
[11] KRISHNAPURAM R, KELLER J M. A possibilistic app-roach to clustering[J]. IEEE Transations on Fuzzy Systems, 1993, 1(2): 98-110.
[12] 但雨芳, 陶剑文, 徐浩特. 可能性聚类假设的半监督分类方法[J].?计算机工程与应用,?2020, 56(9): 65-74.
DAN Y F, TAO J W, XU H T. Semi-supervised classifi-cation method of possibilistic clustering assumption[J]. Com-puter Engineering and Applications, 2020, 56(9): 65-74.
[13] WANG Y Y, CHEN S C. Safety-aware semi-supervised classification[J]. IEEE Transactions on Neural Networks and Learning Systems,2013, 24(11): 1763-1772.
[14] PATEL V M, GOPALAN R, LI R , et al. Visual domain adap-tation: a survey of recent advances[J]. IEEE Signal Proces-sing Magazine, 2015, 32(3): 53-69.
[15] DAN Y F, TAO J W, FU J J, et al. Possibilistic clustering-promoting semi-supervised learning for EEG-based emotion recognition[J]. Frontiers in Neuroscience, 2021, 15: 690044.
[16] TAO J W, DAN Y F. Multi-source co-adaptation for EEG-based emotion recognition by mining correlation informa-tion[J]. Frontiers in Neuroscience, 2021, 15: 677106.
[17] DAN Y F, TAO J W, ZHOU D. Robust multi-source co-adaptation with adaptive loss minimization[J]. Signal Proce-ssing: Image Communication, 2021(5): 116455.
[18] BRUZZONE L, MARCONCINI M. Domain adaptation pro-blems: a DASVM classification technique and a circular validation strategy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(5): 770-787.
[19] XUE H, CHEN S C. Alternative robust local embedding[C]//Proceedings of the 2007 International Conference on Wave-let Analysis and Pattern Recognition, Beijing, Nov 2-4, 2007. Piscataway: IEEE, 2007: 591-596.
[20] BOTTOU L, VAPNIK V. Local learning algorithms[J]. Neural Computation, 1992, 4(6): 888-900.
[21] ATKESON C G，MOORE A W. SCHAAL S. Locally weig-hted learning[J]. Artificial Intelligence Review, 1997, 11： 11-73.
[22] KRISHNAPURAM R，KELLER J M. The possibilistic C-means algorithm: insights and recommendations[J]. IEEE Transactions on Fuzzy Systems, 1996, 4: 385-393.
[23] BELKIN M, NIYOGI P, SINDHWANI V. Manifold regula-rization: a geometric framework for learning from examples[J]. Journal of Machine Learning Research, 2006, 7: 23992434.
[24] MASAYUKI K,?HIROSHI M. Multiple graph label propa-gation by sparse integration[J]. IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(12): 1999-2012.
[25] GENG B, TAO D, XU C, et al. Ensemble manifold regulari-zation[J]. IEEE Transactions on Pattern Analysis and Mac-hine Intelligence, 2012, 34(6): 1227-1233.
[26] GRIFFIN G, HOLUB A, PERONA P. Caltech-256 object category dataset: technical report 7694[R]. California Institute of Technology, 2007.
[27] KULIS B, SAENKO K, DARRELL T. What you saw is not what you get: domain adaptation using asymmetric kernel transforms[C]//Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, Jun 20-25, 2011. Washington: IEEE Computer Society, 2011: 1785-1792.
[28] CHUA T S, TANG J H, HONG R C, et al. NUS-WIDE: a real-world web image database from National University of Singapore[C]//Proceedings of the 8th ACM International Conference on Image and Video Retrieval, Santorini Island, Jul 8-10, 2009. New York: ACM, 2009: 368-375.
[29] YANG J, YAN R, HAUPTMANN A G. Cross-domain video concept detection using adaptive SVMs[C]//Proceedings of the 15th International Conference on Multimedia, Augsburg, Sep 24-29, 2007. New York: ACM, 2007: 188-197.
[30] GEHLER P V, NOWOZIN S. On feature combination for multiclass object classification[C]//Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto,Sep 27-Oct 4, 2009. Washington: IEEE Computer Society, 2009: 221-228.
[31] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[32] TOMMASI T, ORABONA F, CAPUTO B. Learning cate-gories from few examples with multi model knowledge transfer[J]. IEEE Transactions on Pattern Analysis and Mac-hine Intelligence, 2014, 36(5): 928-941.
[33] DUAN L, TSANG I W, XU D. Domain transfer multiple kernel learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(3): 465-479.
[34] LONG M, WANG J, DING G, et al. Adaptation regulariz-ation: a general framework for transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(5): 1076-1089.
[35] DUAN L, XU D, TSANG I W. Domain adaptation from multiple sources: a domain-dependent regularization appro-ach[J]. IEEE Transactions on Neural Networks & Learning Systems, 2012, 23(3): 504-518.
[36] DUAN L X, XU D, CHANG S F. Exploiting web images for event recognition in consumer videos: a multiple source domain adaptation approach[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 1338-1345.
[37] HOU C P, NIE F P, TAO H, et al. Multi-view unsupervised feature selection with adaptive similarity and view weight[J]. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(9): 1998-2011.
[38] TAO J W, ZHOU D, LIU F Y, et al. Latent multi-feature co-regression for visual recognition by discriminatively levera-ging multi-source models[J]. Pattern Recognition, 2019, 87: 296-316.
[39] LONG M S, WANG J. Learning transferable features with deep adaptation networks[J]. arXiv:1502.02791, 2015.
[40] ZHAO Y, SHEN Y, BERNARD A, et al. Evaluation and comparison of current biopsy needle localization and trac-king methods using 3D ultrasound[J]. Ultrasonics, 2017, 73: 206-220.
[41] ZHAO Y, LIEBGOTT H, CACHARD C. Comparison of the existing tool localization methods on two-dimensional ultra-sound images and their tracking results[J]. Control Theory & Applications, 2015, 9(7): 1124-1134.
[42] FU J J, TAO J W. Robust multi-model adaptation regression with local feature space representation[J]. Knowledge-Based Systems, 2019, 174: 160-176.
[43] ROSENSTEIN M T, MARX Z, KAELBLING L P. To trans-fer or not to transfer[C]//Advances in Neural Information Processing Systems 18, 2005.