Deep Learning Method for Fine-Grained Image Categorization

doi:10.3778/j.issn.1673-9418.2103019

Abstract

Abstract:

Fine-grained image categorization aims to distinguish the sub-categories from a certain category of images. Generally, fine-grained data sets have the characteristics of the intra-class similarity and inter-class variation, which makes the task of fine-grained image categorization more challenging. With the increasing development of deep learning, the methods of fine-grained image categorization based on deep learning exhibit more powerful feature representation and generalization capabilities, and can obtain more accurate and stable classification results. Therefore, deep learning has been attracting more and more attentions and research from the researchers in the fine-grained image categorization. In this paper, starting from the background of fine-grained image categorization, the difficulties and the meaning of fine-grained image categorization are introduced. Then, from the perspectives of strong supervision and weak supervision, this paper reviews the research progress of fine-grained image classification algorithms based on deep learning, and a variety of typical classification algorithms with excellent performance are introduced. In addition, the YOLO (you only look once), multi-scale CNN (convolutional neural network), and GAN (generative adversarial networks) model are further discussed in the application of fine-grained image categorization, the perfor-mance of the latest relevant fine-grained data augmentation methods is compared and an analysis of different types of fine-grained categorization methods is made under complex scenarios. Finally, by comparing and summarizing the categorization algorithms, the future improvement directions and challenges are discussed.

Key words: fine-grained image categorization, deep learning, convolutional neural network (CNN), feature extraction

摘要：

细粒度图像分类旨在从某一类别的图像中区分出其子类别，通常细粒度数据集具有类间相似和类内差异大的特点，这使得细粒度图像分类任务更加具有挑战性。随着深度学习的不断发展，基于深度学习的细粒度图像分类方法表现出更强大的特征表征能力和泛化能力，能够获得更准确、稳定的分类结果，因此受到了越来越多研究人员的关注和研究。首先，从细粒度图像分类的研究背景出发，介绍了细粒度图像分类的难点和研究意义。其次，从基于强监督和弱监督两个角度，综述了基于深度学习的细粒度图像分类算法的研究进展，并介绍了多种典型的分类性能优秀的算法。此外，进一步论述了目前关于YOLO、多尺度CNN和生成对抗网络（GAN）等前沿深度学习模型在细粒度图像识别方面的应用，并且对比了最新的相关细粒度图像的数据增强方法的分类效果以及在复杂场景下不同类型的细粒度识别方法的性能特点分析。最后，通过对算法的分类性能进行对比和总结，探讨了未来发展方向和面临的挑战。

关键词: 细粒度图像分类, 深度学习, 卷积神经网络（CNN）, 特征提取

LI Xiangxia, JI Xiaohui, LI Bin. Deep Learning Method for Fine-Grained Image Categorization[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(10): 1830-1842.

李祥霞, 吉晓慧, 李彬. 细粒度图像分类的深度学习方法[J]. 计算机科学与探索, 2021, 15(10): 1830-1842.

References

[1] GE H. Research on image classification method based on deep learning[D]. Chengdu: University of Electronic Science and Technology of China, 2020.
葛昊. 基于深度学习的图像分类方法研究[D]. 成都: 电子科技大学, 2020.
[2] LUO J H, WU J X. A survey on fine-grained image categoriza-tion using deep convolutional features[J]. Acta Automatica Sinica, 2017, 43(8): 1306-1318.
罗建豪, 吴建鑫. 基于深度卷积特征的细粒度图像分类研究综述[J]. 自动化学报, 2017, 43(8): 1306-1318.
[3] LU X Q, JI W J, LI X L, et al. Bidirectional adaptive feature fusion for remote sensing scene classification[J]. Neurocom-puting, 2019, 328: 135-146.
[4] LU X Q, LI X L, MOU L C. Semi-supervised multitask learning for scene recognition[J]. IEEE Transactions on Cybernetics, 2017, 45(9): 1967-1976.
[5] YUAN Y, ZHENG X T, LU X Q. A discriminative represen-tation for human action recognition[J]. Pattern Recognition, 2016, 59: 88-97.
[6] YUAN Y, QI L, LU X Q. Action recognition by joint learning[J]. Image and Vision Computing, 2015, 55: 77-85.
[7] LU X Q, WANG B Q, ZHENG X T, et al. Exploring models and data for remote sensing image caption generation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4): 2183-2195.
[8] VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 652-663.
[9] PENATTI O A B, SILVA F B, VALLE E, et al. Visual word spatial arrangement for image retrieval and classification[J]. Pattern Recognition, 2014, 47(2): 705-720.
[10] LU X Q, CHEN Y X, LI X L. Hierarchical recurrent neural Hashing for image retrieval with hierarchical convolutional features[J]. IEEE Transactions on Image Processing, 2018, 27(1): 106-120.
[11] WAH C, BRANSON S, WELINDER P, et al. The Caltech-UCSD Birds-200-2011 dataset[R]. Pasadena: California Ins-titute of Technology, 2011.
[12] KRAUSE J, STARK M, JIA D, et al. 3D object representa-tions for fine-grained categorization[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops, Sydney, Dec 2-8, 2013. Piscataway: IEEE, 2013: 554-561.
[13] MAJI S, RAHTU E, KANNALA J, et al. Fine-grained visual classification of aircraft[J]. arXiv:1306.5151, 2013.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]// Proceedings of the 26th Annual Conference on Neural Infor-mation Processing Systems, Lake Tahoe, Dec 3-6, 2012. Red Hook: Curran Associates, 2012: 1106-1114.
[15] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[16] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with con-volutions[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 1-9.
[17] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceed-ings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 2818-2826.
[18] SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Menlo Park: AAAI, 2017: 4278-4284.
[19] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learn-ing for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[20] ZHANG N, DONAHUE J, GIRSHICK R B, et al. Part-based R-CNNs for fine-grained category detection[C]//LNCS 8689: Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 834-849.
[21] BRANSON S, VAN HORN G, BELONGIE S, et al. Bird species categorization using pose normalized deep convolu-tional nets[J]. arXiv:1406.2952, 2014.
[22] ZHANG H, XU T, ELHOSEINY M, et al. SPDA-CNN: unifying semantic part detection and abstraction for fine-grained recognition[C]//Proceedings of the 2016 IEEE Con-ference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer So-ciety, 2016: 1143-1152.
[23] UIJLINGS J R R, VAN DE SANDE K E A, GEVERS T, et al. Selective search for object recognition[J]. International Journal of Computer Vision, 2013, 104(2): 154-171.
[24] WEI X S, XIE C W, WU J X. Mask-CNN: localizing parts and selecting descriptors for fine-grained image recognition[J]. arXiv:1605.06878, 2016.
[25] HUANG S L, XU Z, TAO D C, et al. Part-stacked CNN for fine-grained visual categorization[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recogni-tion, Las Vegas, Jun 27-30, 2016. Washington: IEEE Com-puter Society, 2016: 1173-1182.
[26] XIAO T J, XU Y C, YANG K Y, et al. The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Re-cognition, Boston, Jun 7-12, 2015. Washington: IEEE Com-puter Society, 2015: 842-850.
[27] FU J L, ZHENG H L, MEI T. Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 4476-4484.
[28] HE X T, PENG Y X. Weakly supervised learning of part selec-tion model with spatial constraints for fine-grained image classification[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Menlo Park: AAAI, 2017: 4075-4081.
[29] SIMONELLI A, DE NATALE F G B, MESSELODI S, et al. Increasingly specialized ensemble of convolutional neural networks for fine-grained recognition[C]//Proceedings of the 2018 IEEE International Conference on Image Processing, Athens, Oct 7-10, 2018. Piscataway: IEEE, 2018: 594-598.
[30] SUN M, YUAN Y, ZHOU F, et al. Multi-attention multi-class constraint for fine-grained image recognition[J]. arXiv: 1806.05372, 2018.
[31] YANG Z, LUO T, WANG D, et al. Learning to navigate for fine-grained classification[J]. arXiv:1809.00287, 2018.
[32] ZHENG H L, FU J L, MEI T, et al. Learning multi-attention convolutional neural network for fine-grained image recog-nition[C]//Proceedings of the 2017 IEEE International Con-ference on Computer Vision, Venice, Oct 22-29, 2017. Wash-ington: IEEE Computer Society, 2017: 5219-5227.
[33] HUANG C, LI H L, XIE Y R, et al. PBC: polygon-based classifier for fine-grained categorization[J]. IEEE Transac-tions on Multimedia, 2017, 19(4): 673-684.
[34] SHI W W, GONG Y H, TAO X Y, et al. Fine-grained image classification using modified DCNNs trained by cascaded Softmax and generalized large-margin losses[J]. IEEE Tran-sactions on Neural Networks and Learning Systems, 2019, 30(3): 683-694.
[35] HE X T, PENG Y X. Fine-grained image classification via combining vision and language[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recogni-tion, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 7332-7340.
[36] CUI Y, SONG Y, SUN C, et al. Large scale fine-grained ca-tegorization and domain-specific transfer learning[C]//Pro-ceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 4109-4118.
[37] ZHU Y X, LI R C, YANG Y, et al. Learning cascade atten-tion for fine-grained image classification[J]. Neural Networks, 2019, 122: 174-182.
[38] YAN Y C, NI B B, WEI H W, et al. Fine-grained image anal-ysis via progressive feature learning[J]. Neurocomputing, 2020, 396: 254-265.
[39] YAN T T, WANG S J, WANG Z H, et al. Progressive learn-ing for weakly supervised fine-grained classification[J]. Signal Processing, 2020, 171: 107519.
[40] LIN D, SHEN X Y, LU C W, et al. Deep LAC: deep locali-zation, alignment and classification for fine-grained recogni-tion[C]//Proceedings of the 2015 IEEE Conference on Com-puter Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 1666-1674.
[41] ZHANG X P, XIONG H K, ZHOU W G, et al. Picking deep filter responses for fine-grained image recognition[C]//Pro-ceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Wash-ington: IEEE Computer Society, 2016: 1134-1142.
[42] XU Z, HUANG S L, ZHANG Y, et al. Augmenting strong supervision using Web data for fine-grained categorization[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 2524-2532.
[43] YU S Y, WU Y, LI W, et al. A model for fine-grained vehicle classification based on deep learning[J]. Neurocomputing, 2017, 257: 97-103.
[44] HOCHREITER S, SCHMIDHUBER J. Long short-term me-mory[J]. Neural Computing, 1997, 9(8): 1735-1780.
[45] LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNN models for fine-grained visual recognition[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pat-tern Recognition, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 1449-1457.
[46] GAO Y, BEIJBOM O, ZHANG N, et al. Compact bilinear pooling[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 317-326.
[47] GE S Y, GAO Z L, ZHANG B B, et al. Kernelized bilinear CNN models for fine-grained visual recognition[J]. Acta Electronica Sinica, 2019, 47(10): 2134-2141.
葛疏雨, 高子淋, 张冰冰, 等. 基于核化双线性卷积网络的细粒度图像分类[J]. 电子学报, 2019, 47(10): 2134-2141.
[48] KRAUSE J, JIN H L, YANG J C, et al. Fine-grained recogni-tion without part annotations[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recogni-tion, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 5546-5555.
[49] QI L, LU X Q, LI X L. Exploiting spatial relation for fine-grained image classification[J]. Pattern Recognition, 2019, 91: 47-55.
[50] BLOCH I. Fuzzy spatial relationships for image processing and interpretation: a review[J]. Image & Vision Computing, 2005, 23(2): 89-110.
[51] WON C S. Multi-scale CNN for fine-grained image recogni-tion[J].?IEEE Access, 2020, 8: 116663-116674.
[52] XIE E Z, LI G Y, LIU W Y. Improving fine-grained object classification using adversarial generated unlabelled samples [C]//Proceedings of the IEEE 4th International Conference on Multimedia Big Data, Xi??an, Sep 13-16, 2018. Piscata-way: IEEE, 2018: 1-5.
[53] REDMON J, FARHADI A. YOLO9000: better, faster, stronger [J]. arXiv.1612.08242, 2016.
[54] SHE H L, JIE S J, ZHOU J J. 3D-CNN with standard score dimensionality reduction for hyperspectral remote sensing images classification[J]. Computer Engineering and Appli-cations, 2021, 57(4): 169-175.
佘海龙, 解山娟, 邹静洁.?标准分数降维的3D-CNN高光谱遥感图像分类[J].?计算机工程与应用,?2021,?57(4):?169-175.
[55] XU K W, XU B, WU Y, et al. Overview of application of machine learning in ultrasound images[J]. Computer Eng-ineering and Applications, 2021,?57(4):?11-17.
徐可文, 许波, 吴英, 等.?机器学习在超声图像中的应用综述[J].?计算机工程与应用,?2021,?57(4):?11-17.
[56] HU T, QI H, HUANG Q, et al. See better before looking closer: weakly supervised data augmentation network for fine-grained visual classification[J]. arXiv:1901.09891, 2019.
[57] TAN R, YE W J, LIU Y J. Fine-grained image classification based on dual semantic data augmentation and target loca-tion[J/OL]. Computer Engineering (2021-02-08) [2021-09-22]. https://doi.org/10.19678/j.issn.1000-3428.0060111.
谭润, 叶武剑, 刘怡俊. 双语义数据增强及目标定位的细粒度图像分类[J/OL]. 计算机工程(2021-02-08) [2021-09-22]. https://doi.org/10.19678/j.issn.1000-3428.0060111.
[58] DING W Q, YU P F, LI H Y, et al. Weakly supervised fine-grained image classification based on Xception network[J/OL]. Computer Engineering and Applications (2020-12-25) [2021-03-02]. https://kns.cnki.net/kcms/detail/11.2127.TP. 20201225.0921.008.html.
丁文谦, 余鹏飞, 李海燕, 等. 基于Xception网络的弱监督细粒度图像分类[J/OL]. 计算机工程与应用(2020-12-25) [2021-03-02]. https://kns.cnki.net/kcms/detail/11.2127.TP. 20201225.0921.008.html.
[59] LI H, ZHANG X P, TIAN Q, et al. Attribute mix: semantic data augmentation for fine grained recognition[C]//Proceed-ings of the 2020 IEEE International Conference on Visual Communications and Image Processing, Macau, China, Dec 1-4, 2020. Piscataway: IEEE, 2020: 243-246.
[60] YAN C, PANG G S, BAI X, et al. Beyond triplet loss: person re-identification with fine-grained difference-aware pairwise loss[J]. IEEE Transactions on Multimedia, 2021. DOI: 10. 1109/TMM.2021.3069562.
[61] LI W, ZHAO R, XIAO T, et al. DeepReID: deep filter pairing neural network for person re-identification[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pat-tern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 152-159.
[62] ZHENG L, SHEN L Y, TIAN L, et al. Scalable person re-identification: a benchmark[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 1116-1124.
[63] ZHENG Z D, ZHENG L, YANG Y. Unlabeled samples gen-erated by GAN improve the person re-identification base-line in vitro[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017.Washington: IEEE Computer Society, 2017: 3774-3782.
[64] HAN C C, ZHENG R C, GAO C X, et al. Complementation-reinforced attention network for person re-identification[J]. IEEE Transactions on Circuits and Systems, 2020, 30(10): 3433-3445.
[65] XIE P Y, XU X. Multi-scale joint learning for person re-identification[J]. Journal of Beijing University of Aeronau-tics and Astronautics, 2021, 47(3): 613-622.
谢彭宇, 徐新. 基于多尺度联合学习的行人重识别[J]. 北京航空航天大学学报, 2021, 47(3): 613-622.
[66] HAN T T. Fine-grained representations and applications of human action in video understanding[D]. Harbin: Harbin Institute of Technology, 2019.
韩婷婷. 视频理解中人体动作的细粒度表示与应用[D]. 哈尔滨: 哈尔滨工业大学, 2019.
[67] LI K K, LIU L B. Fine-grained image classification model based on bilinear aggregate residual attention[J/OL]. Journal of Frontiers of Computer Science and Technology (2021-02-04) [2021-03-02]. https://kns.cnki.net/kcms/detail/11.5602.TP.20210204.1424.006.html.
李宽宽, 刘立波. 双线性聚合残差注意力的细粒度图像分类模型[J/OL]. 计算机科学与探索(2021-02-04) [2021-03-02]. https://kns.cnki.net/kcms/detail/11.5602.TP.20210204. 1424.006.html.
[68] XIANG Y, FU Y, HUANG H. Global topology constraint network for fine-grained vehicle recognition[J]. IEEE Tran-sactions on Intelligent Transportation Systems, 2020, 21(7): 2918-2929.
[69] FANG J, ZHOU Y, YU Y, et al. Fine-grained vehicle model recognition using a coarse-to-fine convolutional neural net-work architecture[J]. IEEE Transactions on Intelligent Tran-sportation System, 2017, 18(7): 1782-1792.
[70] LI X X, YU L Y, CHANG D L, et al. Dual cross-entropy loss for small-sample fine-grained vehicle classification[J]. IEEE Transactions on Vehicular Technology, 2019, 68(5): 4204-4212.
[71] HUANG Y, LIANG B R, XIE W P, et al. Dual domain multi-task model for vehicle re-identification[J]. IEEE Transactions on Intelligent Transportation System, 2020. DOI: 10.1109/ TITS.2020.3027578.