Survey of Zero-Shot Image Classification

doi:10.3778/j.issn.1673-9418.2010092

Abstract

Abstract:

It is time-consuming and laborious to manually label a large number of samples, and samples from some rare classes are difficult to obtain. Therefore, the zero-shot image classification has become a research hotspot in the computer vision field. Firstly, the zero-shot learning, including direct push zero-shot learning and inductive zero-shot learning, is introduced briefly. Secondly, the space embedding zero-shot image classification methods and the generative model based zero-shot image classification methods with their subclass methods are introduced emphatically. Meanwhile, the mechanism, advantages and disadvantages, and application scenarios of these methods are analyzed and summarized. Thirdly, the main datasets and main evaluation criteria for zero-shot image classification are briefly introduced, and the performance of typical zero-shot image classification methods is compared. Then, the problems such as domain drift, hubness and semantic gap and the corresponding solutions are pointed out. Finally, the future development trends and research hotspots of zero-shot image classification are discussed, such as the accurate location of discriminative region, visual features of high-quality unseen class, generalized zero-shot image classification, etc.

Key words: zero-shot learning, zero-shot image classification, embedding space, generative?model, deep learning

摘要：

面对人工标注大量样本费时费力，一些稀有类别样本难于获取等问题，零样本图像分类成为计算机视觉领域的一个研究热点。首先，对零样本学习，包括直推式零样本学习和归纳式零样本学习进行了简单介绍；其次，重点介绍了基于空间嵌入零样本图像分类方法和基于生成模型零样本图像分类方法以及它们的子类方法，并对这些方法的机制、优缺点和适用场景等进行了分析和总结；然后，简单介绍了零样本图像分类常用数据集和评估方法，并对典型零样本图像分类方法进行了性能比较；接着，指出了现有零样本图像分类中存在的领域漂移、枢纽点和语义鸿沟等问题及相应的解决思路；最后，对零样本图像分类未来发展趋势和研究热点，如判别性区域的准确定位、生成高质量不可见类视觉特征、广义零样本图像分类等进行了探讨。

关键词: 零样本学习, 零样本图像分类, 嵌入空间, 生成模型, 深度学习

LIU Jingyi, SHI Caijuan, TU Dongjing, LIU Shuai. Survey of Zero-Shot Image Classification[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(5): 812-824.

刘靖祎, 史彩娟, 涂冬景, 刘帅. 零样本图像分类综述[J]. 计算机科学与探索, 2021, 15(5): 812-824.

References

[1] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 770-778.
[2] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 2261-2269.
[3] DENG J K, GUO J, XUE N N, et al. ArcFace: additive angular margin loss for deep face recognition[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 4690-4699.
[4] BIEDERMAN I. Recognition-by-components: a theory of human image understanding[J]. Psychological Review, 1987, 94(2): 115-147.
[5] XIAN Y, LAMPERT C H, SCHIELE B, et al. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(9): 2251-2265.
[6] LAROCHELLE H, ERHAN D, BENGIO A Y. Zero-data learning of new tasks[C]//Proceedings of the 23rd National Conference on Artificial Intelligence, Jul 13-17, 2008. Menlo Park: AAAI, 2008: 646-651.
[7] PALATUCCI M, POMERLEAU D, HINTON G E, et al. Zero-shot learning with semantic output codes[C]//Proceedings of the 22nd International Conference on Neural Information Processing Systems, Vancouver, Dec 7-10, 2009. Red Hook: Curran Associates, 2009: 1410-1418.
[8] LAMPERT C H, NICKISCH H, HARMELING S. Learning to detect unseen object classes by between-class attribute transfer[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Piscataway: IEEE, 2009: 951-958.
[9] CHAO W L, CHANGPINYO S, GONG B Q, et al. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild[C]//LNCS 9906: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Berlin, Heidelberg: Springer, 2016: 52-68.
[10] JI Z, WANG H R, YU Y L, et al. A decadal survey of zero-shot image classification[J]. Science in China (Information Sciences), 2019, 49(10): 1299-1320.
冀中, 汪浩然, 于云龙, 等. 零样本图像分类综述: 十年进展[J]. 中国科学: 信息科学, 2019, 49(10): 1299-1320.
[11] FU Y W, HOSPEDALES T M, XIANG T, et al. Attribute learning for understanding unstructured social activity[C]//LNCS 7575: Proceedings of 12th European Conference on Computer Vision, Florence, Nov 7-13, 2012. Berlin, Heidelberg: Springer, 2012: 530-543.
[12] AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for attribute-based classification[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, Jun 23-28, 2013. Piscataway: IEEE, 2013: 819-826.
[13] FROME A, CORRADO G S, SHLENS J, et al. Devise: a deep visual-semantic embedding model[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, Dec 5-8, 2013. Red Hook: Curran Associates, 2013: 2121-2129.
[14] SOCHER R, GANJOO M, MANNING C D, et al. Zero-shot learning through cross-modal transfer[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, Dec 5-8, 2013. Red Hook: Curran Associates, 2013: 935-943.
[15] XIE G S, LIU L, ZHU F, et al. Region graph embedding network for zero-shot learning[C]//LNCS 12349: Proceedings of the European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Berlin, Heidelberg: Springer, 2020: 562-580.
[16] HUYNH D, ELHAMIFAR E. A shared multi-attention framework for multi-label zero-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 8776-8786.
[17] SONG X H, ZENG H T, ZHANG S X, et al. Generalized zero-shot learning with multi-source semantic embeddings for scene recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia, Virtual Event, Seattle, Oct 12-16, 2020. New York: ACM, 2020: 3976-3985.
[18] WANG X L, YE Y F, GUPTA A. Zero-shot recognition via semantic embeddings and knowledge graphs[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 6857-6866.
[19] KAMPFFMEYER M, CHEN Y B, LIANG X D, et al. Rethinking knowledge graph propagation for zero-shot learning [C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 11487-11496.
[20] LIU L, ZHOU T Y, LONG G D, et al. Attribute propagation network for graph zero-shot learning[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 4868-4875.
[21] ZHANG L, XIANG T, GONG S G. Learning a deep embedding model for zero-shot learning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21, 2017. Piscataway: IEEE, 2017: 3010-3019.
[22] SUNG F, YANG Y, ZHANG L, et al. Learning to compare: relation network for few-shot learning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 21-26, 2017. Piscataway: IEEE, 2017: 1199-1208.
[23] LI Y, ZHANG J, ZHANG J, et al. Discriminative learning of latent features for zero-shot recognition[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 7463-7471.
[24] XIE G S, LIU L, JIN X B, et al. Attentive region embedding network for zero-shot learning[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 9384-9393.
[25] LI X, FANG M, LI H K, et al. Zero shot learning based on class visual prototypes and semantic consistency[J]. Pattern Recognition Letters, 2020, 135: 368-374.
[26] WAN Z Y, LI Y, YANG M, et al. Transductive zero-shot learning via visual center adaptation[C]//Proceedings of the 2019 AAAI Conference on Artificial Intelligence, Honolulu, Jan 27-Feb 1, 2019. Menlo Park: AAAI, 2019: 10059-10060.
[27] DEMIREL B, CINBIS R G, IKIZLER-CINBIS N. Learning visually consistent label embeddings for zero-shot learning[C]//Proceedings of the 2019 IEEE International Conference on Image Processing, Taiwan, China, Sep 22-25, 2019. Piscataway: IEEE, 2019: 3656-3660.
[28] HUYNH D, ELHAMIFAR E. Fine-grained generalized zero-shot learning via dense attribute-based attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 4483-4493.
[29] REED S, AKATA Z, LEE H, et al. Learning deep representations of fine-grained visual descriptions[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Piscataway: IEEE, 2016: 49-58.
[30] WANG X S, PANG S M, ZHU J H, et al. Visual space optimization for zero-shot learning[J]. arXiv:1907.00330, 2019.
[31] MIN S, YAO H, XIE H, et al. Domain-specific embedding network for zero-shot recognition[C]//Proceedings of the 27th ACM International Conference on Multimedia, Nice, Oct 21-25, 2019. New York: ACM, 2019: 2070-2078.
[32] LIU Y, GAO X B, GAO Q X, et al. Label-activating framework for zero-shot learning[J]. Neural Networks, 2020, 121: 1-9.
[33] ZHANG L, WANG P, LIU L Q, et al. Towards effective deep embedding for zero-shot learning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(9): 2843-2852.
[34] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Dec 8-13, 2014. Red Hook: Curran Associates, 2014: 2672-2680.
[35] XIAN Y Q, LORENZ T, SCHIELE B, et al. Feature gene-rating networks for zero-shot learning[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 5542-5551.
[36] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[J]. arXiv:1701.07875, 2017.
[37] SARIYILDIZ M B, CINBIS R G. Gradient matching gene-rative networks for zero-shot learning[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 2168-2178.
[38] VERMA V K, BRAHMA D, RAI P. Meta-learning for generalized zero-shot learning[C]//Proceedings of the 2020 AAAI Conference on Artificial Intelligence, New York, Feb 7-12, 2020. Menlo Park: AAAI, 2020: 6062-6069.
[39] MA Y B, XU X, SHEN F M, et al. Similarity preserving feature generating networks for zero-shot learning[J]. Neurocomputing, 2020, 406: 333-342.
[40] LIU H, YAO L, ZHENG Q, et al. Dual-stream generative adversarial networks for distributionally robust zero-shot learning[J]. Information Sciences, 2020, 519: 407-422.
[41] FELIX R, KUMAR V B G, REID I, et al. Multi-modal cycle-consistent generalized zero-shot learning[C]//LNCS 11210: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 21-37.
[42] LI J J, JING M M, LU K, et al. Alleviating feature confusion for generative zero-shot learning[C]//Proceedings of the 27th ACM International Conference on Multimedia, Nice, Oct 21-25, 2019. New York: ACM, 2019: 1587-1595.
[43] CHE T, LI Y R, JACOB A P, et al. Mode regularized generative adversarial networks[J]. arXiv:1612.02136, 2016.
[44] KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[45] MISHRA A, KRISHNA REDDY S, MITTAL A, et al. A generative model for zero shot learning using conditional variational autoencoders[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Piscataway: IEEE, 2018: 2188-2196.
[46] SCHONFELD E, EBRAHIMI S, SINHA S, et al. Generalized zero-and few-shot learning via aligned variational autoencoders[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 8247-8255.
[47] GAO R, HOU X S, QIN J, et al. Zero-VAE-GAN: generating unseen features for generalized and transductive zero-shot learning[J]. IEEE Transactions on Image Processing, 2020, 29: 3665-3680.
[48] ZHANG Z L, LI Y J, YANG J, et al. Cross-layer autoencoder for zero-shot learning[J]. IEEE Access, 2019, 7: 167584-167592.
[49] YU H, LEE B. Zero-shot learning via simultaneous generating and learning[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Dec 8-14, 2019. Red Hook: Curran Associates, 2019: 46-56.
[50] XIAN Y Q, SHARMA S, SCHIELE B, et al. F-VAEGAN-D2: a feature generating framework for any-shot learning[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 10275-10284.
[51] ARDIZZONE L, KRUSE J, WIRKERT S, et al. Analyzing inverse problems with invertible neural networks[J]. arXiv: 1808.04730, 2018.
[52] SHEN Y M, QIN J, HUANG L. Invertible zero-shot recognition flows[J]. arXiv:2007.04873, 2020.
[53] GU Y C, ZHANG L, LIU Y, et al. Generalized zero-shot learning via VAE-conditioned generative flow[J]. arXiv:2009. 00303, 2020.
[54] WAH C, BRANSON S, PERONA P, et al. Multiclass recognition and part localization with humans in the loop[C]//Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Nov 6-13, 2011. Piscataway: IEEE, 2011: 2524-2531.
[55] PATTERSON G, XU C, SU H, et al. The sun attribute database: beyond categories for deeper scene understanding[J]. International Journal of Computer Vision, 2014, 108(1/2): 59-81.
[56] XIAO J X, HAYS J, EHINGER K A, et al. Sun database: large-scale scene recognition from abbey to zoo[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Washington: IEEE Computer Society, 2010: 3485-3492.
[57] FARHADI A, ENDRES I, HOIEM D, et al. Describing objects by their attributes[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Piscataway: IEEE, 2009: 1778-1785.
[58] ZHANG Z M, SALIGRAMA V. Zero-shot learning via semantic similarity embedding[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 7-13, 2015. Piscataway: IEEE, 2015: 4166-4174.
[59] ROMERA-PAREDES B, TORR P H S. An embarrassingly simple approach to zero-shot learning[C]//Proceedings of the 32nd International Conference on Machine Learning, Lille, Jul 6-11, 2015. New York: ACM, 2015: 2152-2161.
[60] KODIROV E, XIANG T, GONG S G. Semantic autoencoder for zero-shot learning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 3174-3183.
[61] AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for image classification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(7): 1425-1438.
[62] LI J J, JING M M, LU K, et al. Leveraging the invariant side of generative zero-shot learning[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 7402-7411.
[63] TONG B, WANG C, KLINKIGT M, et al. Hierarchical disentanglement of discriminative latent features for zero-shot learning[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 11467-11476.
[64] ZHU Y Z, XIE J W, LIU B C, et al. Learning feature-to- feature translator by alternating back-propagation for generative zero-shot learning[C]//Proceedings of the 2019 IEEE International Conference on Computer Vision, Seoul, Oct 27 -Nov 2, 2019. Piscataway: IEEE, 2019: 9843-9853.
[65] YU Y L, JI Z, HAN J G, et al. Episode-based prototype generating network for zero-shot learning[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 14032-14041.