Abstract:

At present, the mainstream 2D convolution neural network method for video action recognition can't extract the relevant information between input frames, which makes it difficult for the network to obtain the spatio-temporal feature information between input frames and improve the recognition accuracy. To solve the existing problems, a universal spatio-temporal feature pyramid module (STFPM) is proposed. STFPM consists of feature pyramid and dilated convolution pyramid, which can be directly embedded into the existing 2D convolution network to form a new action recognition network named spatio-temporal feature pyramid net (STFP-Net). For multi-frame image input, STFP-Net first extracts the individual spatial feature information of each frame input and records it as the original feature. Then, the designed STFPM uses matrix operation to construct the feature pyramid of the original feature. Furthermore, the spatio-temporal features with temporal and spatial correlation are extracted by applying the dilated convolution pyramid to feature pyramid. Next, the original features and spatio-temporal features are fused by a weighted summation and transmitted to the deep network. Finally, the action in the video is classified by full connected layer. Compared with Baseline, STFP-Net introduces negligible additional parameters and computational complexity. Experimental results show that compared with mainstream methods in recent years, STFP-Net has significant improvement in classification accuracy on the general datasets UCF101 and HMDB51.

Key words: action recognition, 2D convolution network, spatio-temporal features, feature pyramid, dilated convolu-tion pyramid

摘要：

目前用于视频行为识别的主流2D卷积神经网络方法无法提取输入帧之间的相关信息,导致网络无法获得输入帧间的时空特征信息进而难以提升识别精度。针对目前主流方法存在的问题,提出了通用的时空特征金字塔模块（STFPM）。STFPM由特征金字塔和空洞卷积金字塔两部分组成,并能直接嵌入到现有的2D卷积神经网络中构成新的行为识别网络——时空特征金字塔网络（STFP-Net）。针对多帧图像输入,STFP-Net首先提取每帧输入的单独空域特征信息,并将这些特征信息记为原始特征;然后,所设计的STFPM利用矩阵转换操作对原始特征构建特征金字塔;其次,利用空洞卷积金字塔对构建的原始特征金字塔提取具有时空关联性的时序特征;接着,将原始特征与时序特征进行加权融合并传递给后续深层网络;最后,利用全连接对网络输出特征进行分类识别。与Baseline相比,STFP-Net引入了可忽略不计的额外参数和计算量。实验结果表明,与近些年主流方法相比,STFP-Net在主流数据库UCF101和HMDB51上的分类准确度具有明显提升。

关键词: 行为识别, 2D卷积网络, 时空特征, 特征金字塔, 空洞卷积金字塔