Deep Optical Flow Learning Networks Combined with Attention Mechanism

doi:10.3778/j.issn.1673-9418.1910052

Abstract

Abstract:

In order to improve the accuracy of deep learning optical flow estimation based on encoder-decoder U-Net, a modified supervised deep optical flow learning network combined with attention mechanism is proposed, which consists of a contracting part and an expanding part. In contracting part, high-level feature information is ex-tracted using a series of convolutional layers, and spatial feature maps are then restored to full resolution by conducting successive deconvolution in expanding part. In this paper, attention mechanism is embedded in U-Net to learn inter-dependencies among the channels so that the channel-wise features can be weighted adaptively, which can enhance the performance of feature extraction. Meanwhile, the proposed network also combines dilated convolution to enlarge the receptive field without changing the size of convolutional kernel. Further, constancy constraints and smoothness constraints from variational method are also adopted so that priori knowledge can be used to improve the accuracy of optical flow estimation. Extensive experiments are conducted on synthesis image sequence datasets and the experi-mental results show the proposed network is effective for improving accuracy of deep learning optical flow estimation.

Key words: optical flow estimation, deep learning, attention mechanism, dilated convolution, prior constraints

摘要：

为提升基于编解码架构的U型网络在深度学习光流估计中的精度，提出了一种结合注意力机制的改进有监督深度学习光流网络。网络由收缩和扩张两部分组成，收缩部分利用一系列卷积层来提取图像之间的高级特征，扩张部分通过反卷积操作将特征图恢复至原始图像分辨率，将通道注意力机制引入U型网络架构中以学习通道之间的相互依赖性，自适应地调整各通道的特征权重，增强网络的特征提取能力。同时，改进的网络还使用了空洞卷积以在卷积核尺寸不变的情况下增大感受野，使用变分光流方法中的恒常约束与平滑约束以进一步利用运动先验知识提升估计效果。最后基于合成图像序列数据集进行了实验验证，实验结果表明所设计的网络能够有效提升深度学习光流估计的准确率。

关键词: 光流估计, 深度学习, 注意力机制, 空洞卷积, 先验约束

ZHOU Haiyun, XIANG Xuezhi, ZHAI Mingliang, ZHANG Rongfang, WANG Shuai. Deep Optical Flow Learning Networks Combined with Attention Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(11): 1920-1929.

周海赟，项学智，翟明亮，张荣芳，王帅. 结合注意力机制的深度学习光流网络[J]. 计算机科学与探索, 2020, 14(11): 1920-1929.

References

[1] Horn B K, Schunck B G. Determining optical flow[J]. Arti-ficial Intelligence, 1981, 17(1): 185-203.
[2] Sun D, Roth S, Black M J. Secrets of optical flow estima-tion and their principles[C]//Proceedings of the 23rd IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, Jun 13-18, 2010. Piscataway: IEEE, 2010: 2432-2439.
[3] Brox T, Malik J. Large displacement optical flow: descriptor matching in variational motion estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(3): 500-513.
[4] Xu W, Liu W, Huang X. Multi-modal self-paced learning for image classification[J]. Neurocomputing, 2018, 309: 134-144.
[5] Dakhia A, Wang T, Lu H. Multi-scale pyramid pooling network for salient object detection[J]. Neurocomputing, 2019, 333: 211-220.
[6] Wang F, Ainouz S, Lian C, et al. Multimodality semantic segmentation based on polarization and color images[J]. Neurocomputing, 2017, 253: 193-200.
[7] Dosovitskiy A, Fischer P, Ilg E, et al. Flownet: learning optical flow with convolutional networks[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Dec 11-18, 2015. Piscataway: IEEE, 2015: 2758-2766.
[8] Vaquero V, Ros G, Moreno-Noguer F, et al. Joint coarse-and-fine reasoning for deep optical flow[C]//Proceedings of the 24th IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 2558-2562.
[9] Ren Z, Yan J, Ni B, et al. Unsupervised deep learning for optical flow estimation[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, Feb 4-9, 2017. Palo Alto: AAAI, 2017: 1495-1501.
[10] Meister S, Hur J, Roth S. UnFlow: unsupervised learning of optical flow with a bidirectional census loss[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans,Feb 2-7, 2018.?Palo Alto: AAAI, 2018: 7251-7259.
[11] Zhu Y, Newsam S. Densenet for dense flow[C]//Procee-dings of the 24th IEEE International Conference on Image Processing, Beijing, Sep 17-20, 2017. Piscataway: IEEE, 2017: 790-794.
[12] Wang Y, Yang Y, Yang Z, et al. Occlusion aware unsuper-vised learning of optical flow[C]//Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Reco-gnition, Salt Lake City, Jun 18-23, 2018. Piscataway: IEEE, 2018: 4884-4893.
[13] Yu J J, Harley A W, Derpanis K G. Back to basics: unsu-pervised learning of optical flow via brightness constancy and motion smoothness[C]//LNCS 9915: Proceedings of the 14th European Conference on Computer Vision,Amsterdam, Oct 8-16, 2016. Berlin, Heidelberg: Springer, 2016: 3-10.
[14] Ranjan A, Black M J. Optical flow estimation using a spatial pyramid network[C]//Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 2720-2729.
[15] Hu P, Wang G, Tan Y. Recurrent spatial pyramid CNN for optical flow estimation[J]. IEEE Transactions on Multimedia, 2018, 20(10): 2814-2823.
[16] Dai J, Huang S, Nguyen T. Pyramid structured optical flow learning with motion cues[C]//Proceedings of the 25th IEEE International Conference on Image Processing, Athens, Oct 7-10, 2018. Piscataway: IEEE, 2018: 3338-3342.
[17] Ilg E, Mayer N, Saikia T, et al. Flownet 2.0: evolution of optical flow estimation with deep networks[C]//Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 1647-1655.
[18] Mayer N, Ilg E, Husser P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Seattle,Jun 27-30, 2016. Piscataway: IEEE, 2016: 4040-4048.
[19] Wang F, Jiang M, Qian C, et al. Residual attention network for image classification[C]//Proceedings of?the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 6450-6458.
[20] Zhang X, Wang T, Qi J, et al. Progressive attention guided recurrent network for salient object detection[C]//Proceedings of?the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Piscat-away: IEEE, 2018: 714-722.
[21] Zhu Y, Zhao C, Guo H, et al. Attention couplenet: fully con-volutional attention coupling network for object detection[J]. IEEE Transactions on Image Processing, 2019, 28(1): 113-126.
[22] Yang F, Yan K, Lu S, et al. Attention driven person re-iden-tification[J]. Pattern Recognition, 2019, 86: 143-155.
[23] Chen L, Yang Y, Wang J, et al. Attention to scale: scale-aware semantic image segmentation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition,Seattle,Jun 27-30, 2016. Piscataway: IEEE, 2016: 3640-3649.
[24] Yuan J J, Zhang L, Chen Y H. Deep neural network based on attention convolution module for image recognition[J]. Computer Engineering and Applications, 2019, 55(8): 9-16.袁嘉杰, 张灵, 陈云华.基于注意力卷积模块的深度神经网络图像识别[J].计算机工程与应用,2019,55(8):9-16.
[25] Sun P, Hu X D, Zhang Y J. Object detection based on deep learning and attention mechanism[J]. Computer Engineering and Applications, 2019, 55(17): 180-184.孙萍, 胡旭东, 张永军.结合注意力机制的深度学习图像目标检测[J].计算机工程与应用,2019,55(17):180-184.
[26] Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[27] Yamashita T, Furukawa H, Fujiyoshi H. Multiple skip con-nections of dilated convolution network for semantic segmen-tation[C]//Proceedings of the 25th IEEE International Con-ference on Image Processing, Athens, Oct 7-10, 2018. Piscat-away: IEEE, 2018: 1593-1597.
[28] Yu F, Koltun V, Funkhouser T. Dilated residual networks[C]//Proceedings of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Piscataway: IEEE, 2017: 636-644.
[29] Shi X B, Dai H L, Zhang D Y, et al. Action prediction method for removing redundant information in optical flow[J]. Journal of Frontiers of Computer Science and Technology, 2019, 13(10): 1745-1753.石祥滨, 代海龙, 张德园, 等.去除光流中冗余信息的动作预测方法[J].计算机科学与探索,2019,13(10):1745-1753.