Abstract: The method of picking frames at regular fixed intervals for video captioning leads to missing much static and dynamic information easily, thereby, the quality of generation sentences can't be further improved. Facing the challenge, this paper proposes a model named dense frame rate sampling based captioning model (DFS-CM). The video is first split into many fragments with fixed length, then, all the frames in each fragment are fed into the pre-trained CNN (convolutional neural network) model for feature. Next, the average or maximum CNN features from each fragment are calculated in line with the corresponding dimension. Additionally, the training strategy of the pre-trained CNN model is improved and the stability and generalization of the model are strengthened. Finally, based on S2VT framework, the model is evaluated with GoogLeNet and ResNet-152 features. The experimental results on Youtube2Text dataset show that the performance is improved greatly compared to the baseline model. Particularly, the B@4 and CIDEr reach to 47.1% and 34.1% respectively with ResNet-152 and maximum mode.

Key words: video, caption generation, GoogLeNet, ResNet, long short term memory (LSTM), dense frame rate sampling

摘要： 使用固定时间间隔取帧的方式用于视频标题生成，易导致多种静态或动态信息丢失，使得生成的句子质量难以提高。针对这一问题，提出了一种使用密集帧率采样的标题生成方法（dense frame rate sampling based captioning model，DFS-CM），将视频分为多个长度统一的片段，提取片段中所有视频帧的深度CNN（convolutional neural network）特征，然后使用均值或最大值方法，降低了特征数量，增强了特征的稀疏程度；同时，还改善了模型的训练策略，增强了模型的稳定性和泛化能力。最后在S2VT框架的基础上，使用GoogLeNet和ResNet-152两种CNN模型，对所提方法进行了验证。在Youtube2Text数据集上的实验结果表明，无论是采用均值特征还是最大值特征，其模型性能与基准模型相比均得到了改善，尤其是使用ResNet-152和最大值方式，其B@4和CIDEr分别达到了47.1%和34.1%。

关键词: 视频, 标题生成, GoogLeNet, ResNet, 长短时记忆（LSTM）, 密集帧率采样