Improved Deep Embedding Clustering with Ensemble Learning

doi:10.3778/j.issn.1673-9418.2007010

Abstract

Abstract:

Recently the rapid development of the deep learning technique has provided a powerful tool for the clustering research, and has given rise to quite a number of deep neural network-based clustering methods. Among these methods, deep embedding clustering (DEC) has been drawing increasing attention, due to its advantage in performing deep representation learning and optimizing clustering assignment simultaneously. However, one limita-tion to DEC lies in its sensitivity to the hyper-parameter λ, which often requires manual fine-tuning. To address this problem, this paper presents an improved deep embedding clustering method with ensemble learning (IDECEL). Instead of searching for a single optimal hyper-parameter, this paper makes use of a set of diversified hyper-parameters λ to construct an ensemble of diversified base clusterings. By exploiting the concept of entropy, this paper evaluates the uncertainty of the clusters in these base clusterings and weights them accordingly. Further, this paper constructs a locally weighted bipartite graph between base clusters and data samples, and efficiently partitions it to obtain a better clustering result. Experimental results on multiple datasets show that the proposed IDECEL method not only alleviates the hyper-parameter sensitivity problem in DEC, but also exhibits more robust clustering performance than several other deep clustering and ensemble clustering methods.

Key words: data clustering, deep clustering, ensemble clustering, ensemble learning, sensitive hyper-parameters

摘要：

近年来深度学习的迅速发展为聚类研究提供了一个有力的工具，并衍生出了许多基于深度神经网络的聚类方法。在这些方法中，深度嵌入聚类（DEC）因其可对深度表示学习和聚类分配同时进行优化的优势而日益受到关注。但是，深度嵌入聚类的一个局限性在于其超参数λ的敏感性，而往往需要诉诸人工调节来解决。对此，提出一种基于集成学习的改进深度嵌入聚类（IDECEL）方法。相较于寻求单个最优超参数的常规做法，提出以多样化超参数λ构建一组具有差异性的基聚类，并结合熵理论对基聚类集合的簇不确定性进行评估与加权，进而在簇与样本之间构建一个局部加权二部图模型，再将之高效划分以得到一个更优聚类结果。在多个数据集上的实验结果表明，提出的IDECEL方法不仅可缓解常规DEC算法超参数敏感性的问题，同时也表现出比其他多个深度聚类和集成聚类方法更为鲁棒的聚类性能。

关键词: 数据聚类, 深度聚类, 集成聚类, 集成学习, 敏感超参数

HUANG Yuxiang, HUANG Dong, WANG Changdong, LAI Jianhuang. Improved Deep Embedding Clustering with Ensemble Learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(10): 1949-1957.

黄宇翔, 黄栋, 王昌栋, 赖剑煌. 基于集成学习的改进深度嵌入聚类算法[J]. 计算机科学与探索, 2021, 15(10): 1949-1957.

References

[1] JAIN A K. Data clustering: 50 years beyond K-means[J]. Pattern Recognition Letters, 2010, 31(8): 651-666.
[2] ELHAMIFAR E, VIDAL R. Sparse subspace clustering: algo-rithm, theory, and applications[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(11): 2765-2781.
[3] FODOR I K. A survey of dimension reduction techniques[R]. Livermore: Lawrence Livermore National Laboratory, 2002: 1-18.
[4] HE X F, CAI D, NIYOGI P. Laplacian score for feature sele-ction[C]//Proceedings of the 18th Advances in Neural Infor-mation Processing Systems, Vancouver, Dec 5-8, 2005. Red Hook: Curran Associates, 2005: 507-514.
[5] HUANG D, CAI X S, WANG C D. Unsupervised feature selec-tion with multi-subspace randomization and collaboration[J]. Knowledge-Based Systems, 2019, 182: 104856.
[6] TIAN F, GAO B, CUI Q, et al. Learning deep representa-tions for graph clustering[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence, Québec, Jul 27-31, 2014. Menlo Park: AAAI, 2014: 1293-1299.
[7] YANG J W, PARIKH D, BATRA D. Joint unsupervised lear-ning of deep representations and image clusters[C]//Procee-dings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 5147-5156.
[8] CHEN G. Deep learning with nonparametric clustering[J]. arXiv:1501.03084, 2015.
[9] PENG X, XIAO S J, FENG J S, et al. Deep subspace cluster-ing with sparsity prior[C]//Proceedings of the 25th Interna-tional Joint Conference on Artificial Intelligence, New York, Jul 9-15, 2016. Menlo Park: AAAI, 2016: 1925-1931.
[10] LIN X, YANG X, LI Y. A deep clustering algorithm based on Gaussian mixture model[J]. Journal of Physics: Conference Series, 2019, 1302(3): 032012.
[11] YANG L X, CHEUNG N M, LI J Y, et al. Deep clustering by Gaussian mixture variational autoencoders with graph embedding[C]//Proceedings of the 2019 IEEE/CVF Interna-tional Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 6439-6448.
[12] XIE J Y, GIRSHICK R B, FARHADI A. Unsupervised deep embedding for clustering analysis[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 478-487.
[13] STREHL A, GHOSH J. Cluster ensembles—a knowledge reuse framework for combining multiple partitions[J]. Journal of Machine Learning Research, 2002, 3: 583-617.
[14] FRED A L N, JAIN A K. Combining multiple clusterings using evidence accumulation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(6): 835-850.
[15] JING L P, TIAN K, HUANG J Z. Stratified feature samp-ling method for ensemble clustering of high dimensional data[J]. Pattern Recognition, 2015, 48(11): 3688-3702.
[16] SHI Q Y, LIANG J Y, ZHAO X W. A clustering ensemble algorithm for incomplete mixed data[J]. Journal of Computer Research and Development, 2016, 53(9): 1979-1989.
史倩玉, 梁吉业, 赵兴旺. 一种不完备混合数据集成聚类算法[J]. 计算机研究与发展, 2016, 53(9): 1979-1989.
[17] HUANG D, LAI J H, WANG C D. Robust ensemble clus-tering using probability trajectories[J]. IEEE Transactions on Knowledge and Data Engineering, 2016, 28(5): 1312-1326.
[18] HUANG D, LAI J, WANG C D. Ensemble clustering using factor graph[J]. Pattern Recognition, 2016, 50: 131-142.
[19] HUANG D, WANG C D, LAI J H. Locally weighted ens-emble clustering[J]. IEEE Transactions on Cybernetics, 2018, 48(5): 1460-1473.
[20] HUANG D, WANG C D, PENG H, et al. Enhanced ensemble clustering via fast propagation of cluster-wise similarities[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021, 51(1): 508-520.
[21] YOUSEFNEZHAD M, HUANG S J, ZHANG D Q. WoCE: a framework for clustering ensemble by exploiting the wisdom of crowds theory[J]. IEEE Transactions on Cybernetics, 2018, 48(2): 486-499.
[22] ZHANG H S, GAO Y K, CHEN Y P, et al. Clustering ensemble algorithm with cluster connection based on wisdom of crowds[J]. Journal of Computer Research and Develop-ment, 2018, 55(12): 2611-2619.
张恒山, 高宇坤, 陈彦萍, 等. 基于群体智慧的簇连接聚类集成算法[J]. 计算机研究与发展, 2018, 55(12): 2611-2619.
[23] HUANG D, WANG C D, WU J S, et al. Ultra-scalable spectral clustering and ensemble clustering[J]. IEEE Transa-ctions on Knowledge and Data Engineering, 2020, 32(6): 1212-1226.
[24] BAI L, LIANG J Y, DU H Y, et al. An information-theoretical framework for cluster ensemble[J]. IEEE Transactions on Knowledge and Data Engineering, 2018, 31(8): 1464-1477.
[25] VINCENT P, LAROCHELLE H, LAJOIE I, et al. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion[J]. Journal of Machine Learning Research, 2010, 11: 3371-3408.
[26] DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9: 2579-2605.
[27] LI Z G, WU X M, CHANG S F. Segmentation using super-pixels: a bipartite graph partitioning approach[C]//Proceed-ings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, Jun 16-21, 2012. Washington: IEEE Computer Society, 2012: 789-796.
[28] GUO X F, LIU X W, ZHU E, et al. Adaptive self-paced deep clustering with data augmentation[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 32(9): 1680-1693.
[29] ASUNCION A, NEWMAN D. UCI machine learning reposi-tory[EB/OL]. [2020-03-10]. http://www.ics.uci.edu/~mlearn/ MLRepository.html.
[30] CHANG J L, WANG L F, MENG G F, et al. Deep adaptive image clustering[C]//Proceedings of the 2017 IEEE Interna-tional Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 5880-5888.
[31] WU J J, LIU H F, XIONG H, et al. K-means-based con-sensus clustering: a unified view[J]. IEEE Transactions on Knowledge and Data Engineering, 2015, 27(1): 155-169.
[32] LIU H F, WU J J, LIU T L, et al. Spectral ensemble clustering via weighted k-means: theoretical and practical evidence[J]. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(5): 1129-1143.
[33] GUO X F, GAO L, LIU X W, et al. Improved deep embed-ded clustering with local structure preservation[C]//Procee-dings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Aug 19-25, 2017: 1753-1759.