Journal of Frontiers of Computer Science and Technology ›› 2021, Vol. 15 ›› Issue (11): 2184-2192.DOI: 10.3778/j.issn.1673-9418.2008027

• Artificial Intelligence • Previous Articles     Next Articles

Task-Aware Dual Prototypical Network for Few-Shot Human-Object Interaction Recognition

AN Ping, JI Zhong, LIU Xiyao   

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
  • Online:2021-11-01 Published:2021-11-09



  1. 天津大学 电气自动化与信息工程学院,天津 300072


Recognizing human-object interaction (HOI) is an important research topic in computer vision. With the great success of deep learning in image classification, the HOI recognition task has also made great progress. However, the problems of instance imbalance and combinatorial explosion still remain the key challenges, which restrict the performance of HOI recognition methods. Therefore, this paper formulates HOI recognition in a few-shot scene to tackle the above problems and proposes a novel task-aware dual prototypical network (TDP-Net) to address few-shot HOI task. Specifically, it first assigns semantic-aware task representations for different tasks as their prior knowledge, subsequently generates attention weights by semantic graph attention module (SGA-Module). It effectively weights the importance on different regions of the visual features, adaptively for different task conditions, which realizes to reason for novel tasks. In addition, it designs a dual prototypes module (DP-Module) to generate both action class prototypes and object class prototypes, which classifies the verb and noun labels respectively. The complex visual relationships between actions and objects can be effectively separated by constructing class prototypes for actions and objects. Meanwhile, owing to the similarity among the related interactions, the knowledge is transferred to the new interactions by reorganizing the action and object prototypes. The experimental results show that the average accuracies of this model outperform the baseline by 3.2 percentage points and 15.7 percentage points on two exper-imental settings, which verifies its effectiveness on the few-shot HOI task.

Key words: computer vision, image classification, human-object interaction (HOI), few-shot learning (FSL), atten-tion mechanism



关键词: 计算机视觉, 图像分类, 人物交互(HOI), 少样本学习(FSL), 注意力机制