Objective Underwater Video Quality Assessment Model via Two-Stream Networks

doi:10.3778/j.issn.1673-9418.2105113

Abstract

Abstract: Underwater videos often suffer from quality degradation. On the one hand, the exponential attenuation of the natural light in water media leads to the loss of underwater video quality. On the other hand, the video is unstable due to the underwater shooting environment such as flow and pressure. Considering the influence of underwater video spatial-temporal features and motion features on video quality, this paper proposes an objective no-reference video quality assessment (VQA) method for underwater scenes, named TS-UVQA (two-stream underwater video quality assessment). TS-UVQA adopts two-stream network structure: Spatial-temporal Net including three-dimensional convolution, switchable normalization and slow fusion strategy is designed to extract spatial-temporal features from the original video frames; Motion Net including two-dimensional convolution and switchable norm-alization is designed to extract motion features from the optical flow fields. Then, decision-level linear fusion is used to aggregate video features and realize high-precision underwater VQA. This paper takes the Pearson linear correlation coefficient (PLCC) and Spearman order correlation coefficient (SROCC) of the video quality assessment result and subjective quality score as indicators. Experimental results show that the motion characteristics extracted from the optical flow can enhance the performance of underwater VQA. Compared with 13 image/video quality assessment methods in an underwater video dataset, TS-UVQA achieves the best performance in terms of PLCC and SROCC. Further experiments on the natural scene video datasets, ECVQ, EVVQ and LIVE achieve a close perfor-mance to the state-of-the-art VQA method, and show that TS-UVQA has good generalization to nature scene VQA.

Key words: quality assessment, no-reference video quality assessment, two-stream network, underwater video

摘要： 水下拍摄的视频存在质量退化效应。一方面，光线在水中传播时呈指数衰减导致水下视频质量损失；另一方面，水下复杂拍摄环境（例如水流等）造成视频的不稳定性。为此，综合考虑水下视频时空特征和运动特征对视频质量的影响，提出一种针对水下场景的客观无参考视频质量评价模型（TS-UVQA）。TS-UVQA采用双流网络结构：设计了由三维卷积、自适应正则化和慢融合策略组成的时空特征提取网络（Spatial-temporal Net），从视频原始帧中学习时空特征；设计了由二维卷积层和自适应正则化堆叠的运动特征提取网络（Motion Net），从光流场块中学习水下视频的相关运动特征；使用决策级融合实现高精度的水下视频质量评价。以模型的视频质量评价结果与主观质量分数的皮尔森线性相关系数（PLCC）和斯皮尔曼秩序相关系数（SROCC）为指标，通过实验验证了TS-UVQA中运动网络对于水下视频质量评价的性能提升效果，同时与13种图像和视频客观质量评价方法相比，在水下视频数据集上取得最佳性能。此外，TS-UVQA在3个自然场景视频数据集（ECVQ、EVVQ、LIVE）也取得了与最先进方法接近的相关系数，表明方法具有良好的泛化性能。

关键词: 质量评价, 无参考视频质量评价, 双流法, 水下视频

SONG Wei, XIAO Yi, DU Yanling, ZHANG Minghua. Objective Underwater Video Quality Assessment Model via Two-Stream Networks[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(2): 409-418.

宋巍, 肖毅, 杜艳玲, 张明华. 双流网络的水下视频客观质量评价模型[J]. 计算机科学与探索, 2023, 17(2): 409-418.

References

[1] CHIANG J Y, CHEN Y. Underwater image enhancement by wavelength compensation and dehazing[J]. IEEE Transactions on Image Processing, 2012, 21(4): 1756-1769.
[2] SERIES B T. Methodology for the subjective assessment of the quality of television pictures[R]. Recommendation ITU-R BT, 2012.
[3] SCHETTINI R, CORCHS S. Underwater image processing: state of the art of restoration and image enhancement methods[J]. EURASIP Journal on Advances in Signal Processing, 2010: 746052.
[4] SAAD M A, BOVIK A C, CHARRIER C. Blind prediction of natural video quality[J]. IEEE Transactions on Image Processing, 2014, 23(3): 1352-1365.
[5] XU J T, YE P, LIU Y, et al. No-reference video quality assessment via feature learning[C]//Proceedings of the 2014 IEEE International Conference on Image Processing, Paris, Oct 27-30, 2014. Piscataway: IEEE, 2014: 491-495.
[6] MEN H, LIN H H, SAUPE D. Spatiotemporal feature combination model for no-reference video quality assessm-ent[C]//Proceedings of the 10th International Conference on Quality of Multimedia Experience, Cagliari, May 29-Jun 1, 2018. Piscataway: IEEE, 2018: 1-3.
[7] KANG L, YE P, LI Y, et al. Convolutional neural networks for no-reference image quality assessment[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 1733-1740.
[8] JIA H Z, SUN Q S, WANG T H. PCANet for blind image quality assessment[C]//Proceedings of the 11th International Conference on Computational Intelligence and Security, Shenzhen, Dec 19-20, 2015. Washington: IEEE Computer Society, 2015: 195-198.
[9] BIANCO S, CELONA L, NAPOLETANO P, et al. On the use of deep learning for blind image quality assessment[J]. Signal, Image and Video Processing, 2018, 12(2): 355-362.
[10] YAN Q, GONG D, ZHANG Y. Two-stream convolutional networks for blind image quality assessment[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2200-2211.
[11] LI C, BOVIK A C, WU X. Blind image quality assessment using a general regression neural network[J]. IEEE Transa-ctions on Neural Networks, 2011, 22(5): 793-799.
[12] LIU W T, DUANMU Z F, WANG Z. End-to-end blind quality assessment of compressed videos using deep neural networks[C]//Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Oct 22-26, 2018. New York: ACM, 2018: 546-554.
[13] VARGA D, SZIRáNYI T. No-reference video quality asse-ssment via pretrained CNN and LSTM networks[J]. Signal, Image and Video Processing, 2019, 13(8): 1569-1576.
[14] LI D Q, JIANG T T, JIANG M. Quality assessment of in-the-wild videos[C]//Proceedings of the 27th ACM Internat-ional Conference on Multimedia, Nice, Oct 21-25, 2019. New York: ACM, 2019: 2351-2359.
[15] SCHECHNER Y Y, KARPEL N. Recovery of underwater visibility and structure by polarization analysis[J]. IEEE Journal of Oceanic Engineering, 2005, 30 (3): 570-587.
[16] HOU W, WEIDEMANN A D. Objectively assessing under-water image quality for the purpose of automated restor-ation[C]//Proceedings of the Visual Information Processing XVI, Orlando, Apr 10, 2007. San Francisco: SPIE, 2007: 65750Q.
[17] YANG M, SOWMYA A. An underwater color image quality evaluation metric[J]. IEEE Transactions on Image Proce-ssing, 2015, 24(12): 6062-6071.
[18] PANETTA K, GAO C, AGAIAN S. Human-visual-system-inspired underwater image quality measures[J]. IEEE Journal of Oceanic Engineering, 2016, 41(3): 541-551.
[19] MORENO-ROLDáN J, PONCELA J, OTERO P, et al. A no-reference video quality assessment model for underwater networks[J]. IEEE Journal of Oceanic Engineering, 2018, 45(1): 342-346.
[20] 郭继昌, 李重仪, 张艳, 等. 面向水下图像的质量评价方法[J]. 中国图象图形学报, 2017, 22(1): 1-8.
GUO J C, LI C Y, ZHANG Y, et al. Quality assessment method for underwater images[J]. Journal of Image and Graphics, 2017, 22(1) : 1-8.
[21] 宋巍, 刘诗梦, 黄冬梅, 等. 适用小样本的无参考水下视频质量评价方法[J]. 中国图象图形学报, 2020, 25(9): 1787-1799.
SONG W, LIU S M, HUANG D M, et al. Non-reference underwater video quality assessment method for small size samples[J]. Journal of Image and Graphics, 2020, 25(9): 1787-1799.
[22] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[J]. arXiv:1406. 2199, 2014.
[23] BROX T, BREGLER C, MALIK J. Large displacement optical flow[C]//Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recog-nition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 41-48.
[24] LUO P, REN J, PENG Z, et al. Differentiable learning-to-normalize via switchable normalization[J]. arXiv:1806.10779, 2018.
[25] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning, Lille, Jul 6-11, 2015: 448-456.
[26] ULYANOV D, VEDALDI A, LEMPITSKY V. Instance normalization: the missing ingredient for fast stylization[J]. arXiv:1607.08022, 2016.
[27] BA J L, KIROS J R, HINTON G E. Layer normalization[J]. arXiv:1607.06450, 2016.
[28] KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural net-works[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 1725-1732.
[29] RIMAC-DRLJE S, VRANJE? M, ?AGAR D. Foveated mean squared error—a novel video quality metric[J]. Multimedia Tools and Applications, 2010, 49(3): 425-445.
[30] VRANJE? M, RIMAC-DRLJE S, GRGI? K. Review of objective video quality metrics and performance comparison using different databases[J]. Signal Processing: Image Communication, 2013, 28(1): 1-19.
[31] SESHADRINATHAN K, SOUNDARARAJAN R, BOVIK A C, et al. A subjective study to evaluate video quality assessment algorithms[C]//Proceedings of the Human Vision and Electronic Imaging XV, San Jose, Jan 18-21, 2010. San Francisco: SPIE, 2010: 75270.
[32] SESHADRINATHAN K, SOUNDARARAJAN R, BOVIK A C, et al. Study of subjective and objective quality assess-ment of video[J]. IEEE Transactions on Image Processing, 2010, 19(6): 1427-1441.
[33] AGAIAN S S, SILVER B, PANETTA K. Transform coef-ficient histogram-based image enhancement algorithms using contrast entropy[J]. IEEE Transactions on Image Proces-sing, 2007, 16(3): 741-758.
[34] MITTAL A, MOORTHY A K, BOVIK A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708.
[35] MITTAL A, SAAD M A, BOVIK A C. A completely blind video integrity oracle[J]. IEEE Transactions on Image Proc-essing, 2015, 25(1): 289-300.