计算机科学与探索 ›› 2022, Vol. 16 ›› Issue (11): 2415-2429.DOI: 10.3778/j.issn.1673-9418.2205003

• 综述·探索 • 上一篇    下一篇

手语识别与翻译综述

闫思伊1, 薛万利1,+(), 袁甜甜2   

  1. 1.天津理工大学 计算机科学与工程学院,天津 300384
    2.天津理工大学 聋人工学院,天津 300384
  • 收稿日期:2022-05-05 修回日期:2022-07-20 出版日期:2022-11-01 发布日期:2022-11-16
  • 通讯作者: + E-mail: xuewanli@email.tjut.edu.cn
  • 作者简介:闫思伊(1996—),女,河南驻马店人,硕士研究生,主要研究方向为手语识别、手语翻译。
    薛万利(1986—),男,江苏南京人,博士,CCF会员,主要研究方向为目标跟踪、手语识别。
    袁甜甜(1980—),女,天津人,博士,教授,CCF会员,主要研究方向为手语识别、计算机网络。
  • 基金资助:
    国家自然科学基金(61906135);国家自然科学基金(62020106004);国家自然科学基金(92048301)

Survey of Sign Language Recognition and Translation

YAN Siyi1, XUE Wanli1,+(), YUAN Tiantian2   

  1. 1. School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China
    2. Technical College for the Deaf, Tianjin University of Technology, Tianjin 300384, China
  • Received:2022-05-05 Revised:2022-07-20 Online:2022-11-01 Published:2022-11-16
  • About author:YAN Siyi, born in 1996, M.S. candidate. Her research interests include sign language recognition and sign language translation.
    XUE Wanli, born in 1986, Ph.D., member of CCF. His research interests include target tracking and sign language recognition.
    YUAN Tiantian, born in 1980, Ph.D., professor, member of CCF. Her research interests include sign language recognition and computer network.
  • Supported by:
    National Natural Science Foundation of China(61906135);National Natural Science Foundation of China(62020106004);National Natural Science Foundation of China(92048301)

摘要:

不同于有声语言,手语主要由连续的手势动作构成。手语识别与翻译是促成听障人士与健听人士之间无障碍交流的重要手段。手语识别与翻译研究任务通过对手语视频进行处理分析并以文字形式显示识别结果,是一种典型的多领域交叉研究。近年来,基于深度学习的手语识别与翻译研究获得了长足的进步。为了便于研究者们系统、全面地了解手语识别与翻译研究任务,分别以手语识别和手语翻译两大任务为主线,从三方面展开综述工作:首先,对具备代表性的手语识别和手语翻译研究工作进行分类总结并分析其特点;其次,归纳整理当前常用的不同国别手语识别与翻译研究数据集,分别从孤立词和连续手语语句两个角度进行分类,同时根据手语识别和手语翻译研究任务的差异性,介绍了对应的评价指标体系;最后,从手语视觉特征的有效信息提取、多线索权重分配、手语与自然语言语法对应及手语数据集资源等方面总结了手语识别与翻译研究目前存在的主要挑战。

关键词: 手语识别, 手语翻译, 深度学习, 神经网络

Abstract:

Different from spoken languages, sign language is mainly composed of continuous gestures. Sign langu-age recognition and translation are important means of facilitating barrier-free communication between the hearing-impaired and the hearing person. The sign language recognition and translation research task is a typical multi-domain cross-study by processing and analyzing sign language videos and displaying the recognition results in text form. In recent years, sign language recognition and translation research based on deep learning has made great progress. In order to facilitate researchers to systematically and comprehensively understand the research tasks of sign language recognition and translation, the review work is carried out from the perspectives of sign language recognition and sign language translation. Firstly, the translation research work is classified and summarized and its characteristics are analyzed. Secondly, the common sign language recognition and translation research datasets of different countries are summarized and classified from the perspectives of isolated sign language words and continuous sign language sentences. Based on the difference in research tasks, the corresponding evaluation index system is introduced. Finally, the major challenges of current research on sign language recognition and translation are summarized from the aspects of effective information extraction of sign language visual features, multi-cue weight assignment, relationship between sign language and natural language grammar, and sign language dataset resources.

Key words: sign language recognition, sign language translation, deep learning, neural network

中图分类号: