Deep Clustering Algorithm Based on Denoising and Self-Attention

doi:10.3778/j.issn.1673-9418.2006068

Abstract

Abstract:

Recently, deep clustering methods have achieved perfect clustering performances, which simultaneously perform clusters assignment and features representation learning. However, the performances greatly degenerate with the decreasing of images quality such as noisy image. To this end, a novel deep clustering method DDC (deep denoising clustering) is proposed. A deep convolutional denoising auto-encoder is employed to learn the robust features representation from noisy image, and self-attention mechanism improves the ability of capturing local features. End-to-end jointly training obtains features more suitable to clustering tasks and then completes clustering assignment. The similarities between feature embeddings and cluster-centers are weighted by different coefficients to enlarge the differences between intra-clusters and inter-clusters. The experimental results on the public datasets demonstrate that the proposed DDC can provide better clustering performances. And compared with other deep clustering algorithms, for example, the clustering accuracy of DDC is 0.803 while DEC (deep embedding clustering) is 0.597 on the COIL-20 dataset. Overall, DDC algorithm with the help of deep convolutional denoising auto-encoder and self-attention can efficiently group noisy images, and further enlarges the application range of deep clustering analysis.

Key words: deep clustering, feature representation, denoising convolutional auto-encoder, self-attention mechanism, clustering accuracy

摘要：

近几年，联合聚类划分和表示学习的深度聚类方法提供了出色的聚类性能，但随着图像质量的下降（比如噪声图像），聚类结果还不能令人满意。为此，提出一种新的深度聚类算法（DDC）。深度卷积降噪自编码器学习噪声数据的特征表示；自注意力机制提高网络捕获局部关键信息的能力；端到端的联合训练得到合适的特征表示并完成聚类分配；对数据点和类中心的相似度赋予不同的权重，扩大同类和异类之间的差异。在公开图像数据集上的实验表明DDC算法的聚类性能更高；并与其他深度聚类算法进行对比，例如在COIL-20上DDC的聚类精度是0.803，而DEC算法仅是0.597。总之，结合自注意力和深度卷积降噪自编码器的DDC算法能对噪声图像进行更有效的聚类分析，扩大了图像聚类的应用范围。

关键词: 深度聚类, 特征表示, 卷积降噪自编码器, 自注意力机制, 聚类精度

CHEN Junfen, ZHANG Ming, ZHAO Jiacheng, XIE Bojun, LI Yan. Deep Clustering Algorithm Based on Denoising and Self-Attention[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(9): 1717-1727.

陈俊芬, 张明, 赵佳成, 谢博鋆, 李艳. 结合降噪和自注意力的深度聚类算法[J]. 计算机科学与探索, 2021, 15(9): 1717-1727.

References

[1] LUXBURG U V. A tutorial on spectral clustering[J]. Statistics & Computing, 2007, 17(4): 395-416.
[2] JAIN A K. Data clustering: 50 years beyond K-means[J]. Pattern Recognition Letters, 2010, 31(8): 651-666.
[3] MASCI J, MEIER U, CIRESAN D C, et al. Stacked convo-lutional auto-encoders for hierarchical feature extraction[C]//LNCS 6791: Proceedings of the 21st International Conference on Artificial Neural Networks, Espoo, Jun 14-17, 2011. Berlin, Heidelberg: Springer, 2011: 52-59.
[4] RIFAI S, VINCENT P, MULLER X, et al. Contractive auto-encoders: explicit invariance during feature extraction[C]//Proceedings of the 28th International Conference on Machine Learning, Bellevue, Jun 28-Jul 2, 2011. Madison: Omni Press, 2011: 833-840.
[5] BALDI P. Autoencoders, unsupervised learning and deep archi-tectures[C]//Proceedings of the International Conference on Unsupervised and Transfer Learning Workshop, Bellevue, Jul 2, 2011: 37-50.
[6] YUAN F N, ZHANG L, SHI J T, et al. Theories and applica-tions of auto-encoder neural networks: a literature survey[J]. Chinese Journal of Computers, 2019, 42(1): 203-230.
袁非牛, 章琳, 史劲亭, 等. 自编码神经网络理论及应用综述[J]. 计算机学报, 2019, 42(1): 203-230.
[7] WAN J, WU F, HE Y B, et al. Clustering algorithm for high-dimensional data under new dimensionality reduction criteria[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(1): 96-107.
万静, 吴凡, 何云斌, 等. 新的降维标准下的高维数据聚类算法[J]. 计算机科学与探索, 2020, 14(1): 96-107.
[8] TIAN F, GAO B, CUI Q, et al. Learning deep representations for graph clustering[C]//Proceedings of the 28th AAAI Con-ference on Artificial Intelligence, Québec City, Jul 27-31, 2014. Menlo Park: AAAI, 2014: 1293-1299.
[9] HUANG P, HUANG Y, WANG W, et al. Deep embedding network for clustering[C]//Proceedings of the 22nd Interna-tional Conference on Pattern Recognition, Stockholm, Aug 24-28, 2014. Washington: IEEE Computer Society, 2014: 1532-1537.
[10] XIE J Y, GIRSHICK R, FARHADI A. Unsupervised deep embedding for clustering analysis[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 478-487.
[11] ZHENG H, FU J, MEI T, et al. Learning multi-attention convolutional neural network for fine-grained image reco-gnition[C]//Proceedings of the 2017 IEEE International Con-ference on Computer Vision, Venice, Oct 22-29, 2017. Wa-shington: IEEE Computer Society, 2017: 5219-5227.
[12] FU J, ZHENG H, MEI T. Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 4438-4446.
[13] VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning, Helsinki, Jul 5-9, 2008: 1096-1103.
[14] RUMELHART D E, HINTON G E, WILLIAMS R J. Lear-ning representations by back-propagating errors[J]. Nature, 1986, 323: 533-536.
[15] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313: 504-507.
[16] CHEN M M, XU Z X, WEINBERGER K, et al. Marginalized denoising autoencoders for domain adaptation[C]//Proceedings of the 29th International Conference on Machine Learning, Edinburgh, Jun 26-Jul 1, 2012. Madison: Omni Press, 2012: 1-8.
[17] CARON M, BOJANOWSKI P, JOULIN A, et al. Deep clustering for unsupervised learning of visual features[C]//LNCS 11218: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Hei-delberg: Springer, 2018: 132-149.
[18] CHANG J, WANG L, MENG G, et al. Deep adaptive image clustering[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 5880-5888.
[19] LI F F, QIAO H, ZHANG B, et al. Discriminatively boosted image clustering with fully convolutional auto-encoders[J]. Pattern Recognition, 2018, 83: 161-173.
[20] ZHANG S, GONG Y H, WANG J J. The development of deep convolution neural network and its applications on com-puter vision[J]. Chinese Journal of Computers, 2019, 42(3): 453-482.
张顺, 龚怡宏, 王进军. 深度卷积神经网络的发展及其在计算机视觉领域的应用[J]. 计算机学报, 2019, 42(3): 453-482.
[21] DIZAJI K G, HERANDI A, DENG C, et al. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 5736-5745.
[22] YANG J W, PARIKH D, BATRA D. Joint unsupervised learning of deep representations and image clusters[C]//Pro-ceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Washington: IEEE Computer Society, 2016: 5147-5156.
[23] XIE J Y, GAO H C, XIE W X. K-nearest neighbors optimized clustering algorithm by fast search and finding the density peaks of a dataset[J]. Scientia Sinica Informationis, 2016, 46(2): 258-280.
谢娟英, 高红超, 谢维信. K近邻优化的密度峰值快速搜索聚类算法[J]. 中国科学: 信息科学, 2016, 46(2): 258-280.
[24] XIE J Y, GAO H C, XIE W, et al. Robust clustering by detecting density peaks and assigning points based on fuzzy weighted k-nearest neighbors[J]. Information Sciences, 2016, 354: 19-40.
[25] KINGMA D P, BA L. Adam: a method for stochastic opti-mization[C]//Proceedings of the 3rd International Conference on Learning Representations, San Diego, May 7-9, 2015: 1-15.
[26] RUDER S. An overview of gradient descent optimization algorithms[J]. arXiv:1609.04747, 2016.