Journal of Frontiers of Computer Science and Technology ›› 2022, Vol. 16 ›› Issue (10): 2234-2248.DOI: 10.3778/j.issn.1673-9418.2112080

• Surveys and Frontiers • Previous Articles     Next Articles

Review of Image Captioning Methods Based on Encoding-Decoding Technology

GENG Yaogang, MEI Hongyan+(), ZHANG Xing, LI Xiaohui   

  1. School of Electronic and Information Engineering, Liaoning University of Technology, Jinzhou, Liaoning 121000, China
  • Received:2021-12-20 Revised:2022-04-19 Online:2022-10-01 Published:2022-10-14
  • About author:GENG Yaogang, born in 1997, M.S. candidate.His research interests include natural language processing and image captioning.
    MEI Hongyan, born in 1978, Ph.D., associate professor, member of CCF. Her research interests include data mining, big data analysis and network services.
    ZHANG Xing, born in 1975, Ph.D., professor, member of CCF. His research interests include network architecture and protocol and information security.
    LI Xiaohui, born in 1978, Ph.D., associate professor. Her research interests include network security, privacy protection and cloud computing.
  • Supported by:
    National Natural Science Foundation for Youth of China(61802161);Scientific Research Project of Liaoning Provincial Department of Education(JZL202015404);Scientific Research Project of Liaoning Provincial Department of Education(LJKZ0625);General Project of Liaoning Provincial Department of Education(LJKZ0618)

编码-解码技术的图像标题生成方法研究综述

耿耀港, 梅红岩+(), 张兴, 李晓会   

  1. 辽宁工业大学 电子与信息工程学院,辽宁 锦州 121000
  • 通讯作者: + E-mail: 715014795@qq.com
  • 作者简介:耿耀港(1997—),男,山东济宁人,硕士研究生,主要研究方向为自然语言处理、图像标题生成。
    梅红岩(1978—),女,辽宁葫芦岛人,博士,副教授,CCF会员,主要研究方向为数据挖掘、大数据分析、网络服务。
    张兴(1975—),男,辽宁葫芦岛人,博士,教授,CCF专业会员,主要研究方向为网络体系架构与协议、信息安全。
    李晓会(1978—),女,辽宁盘锦人,博士,副教授,主要研究方向为网络安全、隐私保护、云计算。
  • 基金资助:
    国家自然科学基金青年项目(61802161);辽宁省教育厅科学研究项目(JZL202015404);辽宁省教育厅科学研究项目(LJKZ0625);辽宁省教育厅面上项目(LJKZ0618)

Abstract:

In recent years, image caption generation, as a multimodal task in the field of artificial intelligence, integrates the related research of computer vision and natural language processing, and can realize the modal conversion from image to text. It plays an important role in visual assistance and image understanding, and has attracted extensive attention from researchers. Firstly, this paper describes the task of image caption generation, and introduces three image caption generation methods: template-based method, retrieval-based method and encode-decode method. Their respective method ideas, representative research and advantages and disadvantages are also introduced. Secondly, from the model structure, the research progress of image understanding phase and caption generation phase, this paper expounds in detail the method based on encoding-decoding, and summarizes the research over years into the research of image understanding and caption generation. Image understanding research includes attention mechanism and semantic aspects. The research of caption generation is divided into traditional caption generation, dense caption generation and stylish caption generation. The performance, advantages and disadvantages of the model are summarized, and the datasets and evaluation index of the performance evaluation of the image captioning model are introduced. Finally, the challenges and difficulties in the field of image captioning are pointed out.

Key words: image caption generation, encode, decode, multimodal, attention mechanism

摘要:

近年来,图像标题生成作为人工智能领域中的多模态任务,融合了计算机视觉和自然语言处理的相关研究,能够实现从图像到文本的模态转换,在视觉辅助和图像理解等方面有着重要作用,备受研究者们的广泛关注。首先对图像标题生成任务进行了阐述,介绍了三种图像标题生成方法,基于模板的方法、基于检索的方法和基于编码-解码的方法以及各自的方法思路、代表性研究和优缺点。其次从方法的模型构成、图像理解阶段和标题生成阶段的研究进展等方面对基于编码-解码的方法进行了详细阐述。将近年来的研究总结归纳为图像理解方面的研究和标题生成方面的研究,其中图像理解方面的研究包括注意力机制的研究和语义获取方面的研究,标题生成方面的研究分为传统标题、密集标题和个性化标题生成的研究,并总结了模型性能及优缺点,介绍了图像标题生成模型进行性能评估的数据集和评测指标。最后指出图像标题生成领域研究面对的挑战和难点。

关键词: 图像标题生成, 编码, 解码, 多模态, 注意力机制

CLC Number: