Review of Research on Video Quality Assessment Based on Deep Learning

doi:10.3778/j.issn.1673-9418.2008009

Abstract

Abstract:

Video quality assessment (VQA) is based on the subjective quality assessment results of the human eye, using models to evaluate distorted videos. It is difficult for traditional assessment methods to make subjective assessment results consistent with objective assessment results. The VQA methods based on deep learning can be evaluated through independent learning of the model without adding manual features. It is of great significance to video quality monitor and assessment, and has become one of the research hotspots in computer vision. First, this paper introduces the research background and main research methods of VQA. Second, it introduces the VQA methods based on deep learning from both the full-reference and no-reference ones, and uses the convolutional neural network model to classify and compare the no-reference evaluation methods. Then it introduces the related databases and performance evaluation indices of the VQA models and compares the algorithm performance. Finally, this paper summarizes the existing problems in the current VQA researches and looks forward to the challenges and development directions in this field.

Key words: deep learning, video quality assessment (VQA), objective evaluation, no-reference, convolutional neural network (CNN)

摘要：

视频质量评价（VQA）是以人眼的主观质量评估结果为依据，使用算法模型对失真视频进行评估。传统的评估方法难以做到主观评价结果与客观评价结果相一致。基于深度学习的视频质量评价方法无需加入手工特征，通过模型自主学习即可进行评估，对视频质量的监控和评价有重要意义，已成为计算机视觉领域的研究热点之一。首先对视频质量评价的研究背景和主要研究方法进行介绍；其次从全参考型和无参考型两方面介绍基于深度学习的客观质量评价方法，并且从所用的卷积神经网络模型对无参考型评价方法进行了分类比较；接着介绍视频质量评价算法的相关数据库和评价算法性能指标，并对算法性能进行比较；最后对目前视频质量评价研究存在的问题进行总结，并展望了该领域面临的挑战和未来发展方向。

关键词: 深度学习, 视频质量评价（VQA）, 客观评价, 无参考, 卷积神经网络（CNN）

TAN Yaya, KONG Guangqian. Review of Research on Video Quality Assessment Based on Deep Learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(3): 423-437.

谭娅娅, 孔广黔. 基于深度学习的视频质量评价研究综述[J]. 计算机科学与探索, 2021, 15(3): 423-437.

References

[1] ZHU X C, TANG G J. Overview of visual quality assessment methods[J]. Journal of Nanjing University of Posts and Tele-communications (Natural Science Edition), 2018, 38(4): 1-11.
朱秀昌, 唐贵进. 可视质量评价方法综述[J]. 南京邮电大学学报(自然科学版), 2018, 38(4): 1-11.
[2] WANG Z M. Review of no-reference image quality assess-ment[J]. Acta Automatica Sinica, 2015, 41(6): 1062-1079.
王志明. 无参考图像质量评价综述[J]. 自动化学报, 2015, 41(6): 1062-1079.
[3] SUCHOW J W, ALVAREZ G A. Motion silences awareness of visual change[J]. Current Biology, 2011, 21(2): 140-143.
[4] SESHADRINATHAN K, BOVIK A C. Motion tuned spatio-temporal quality assessment of natural videos[J]. IEEE Trans-actions on Image Processing, 2010, 19(2): 335-350.
[5] VU P V, VU C T, CHANDLER D M. A spatiotemporal most-apparent-distortion model for video quality assessment[C]//Proceedings of the 18th IEEE International Conference on Image Processing, Brussels, Sep 11-14, 2011. Piscataway: IEEE, 2011: 2505-2508.
[6] TAGLIASACCHI M, VALENZISE G, NACCARI M, et al. A reduced-reference structural similarity approximation for videos corrupted by channel errors[J]. Multimedia Tools & Applications, 2010, 48(3): 471-492.
[7] XU J T, YE P, LIU Y, et al. No-reference video quality ass-essment via feature learning[C]//Proceedings of the 2014 IEEE International Conference on Image Processing, Paris, Oct 27-30, 2014. Piscataway: IEEE, 2014: 491-495.
[8] QIAN J S, WU D, Li L D, et al. Image quality assessment based on multi-scale representation of structure[J]. Digital Signal Processing, 2014, 33(3): 125-133.
[9] MOORTHY A K, BOVIK A C. A two-step framework for constructing blind image quality indices[J]. IEEE Signal Pro-cessing Letters, 2010, 17(5): 513-516．
[10] MITTAL A, SAAD M, BOVIK A C. Assessment of video naturalness using time-frequency statistics[C]//Proceedings of the 2014 IEEE International Conference on Image Processing, Paris, Oct 27-30, 2014. Piscataway: IEEE, 2014: 571-574.
[11] SAAD M A, BOVIK A C, CHARRIER C. Blind prediction of natural video quality[J]. IEEE Transactions on Image Pro-cessing, 2014, 23(3): 1352-1365.
[12] GALKANDAGE C, CALIC J, DOGAN S, et al. Full-reference stereoscopic video quality assessment using a motion sensi-tive HVS model[J]. IEEE Transactions on Circuits and Sys-tems for Video Technology, 2020: 1-1.
[13] REDDY D S V, CHANNAPPAYYA S S. No-reference video quality assessment using natural spatiotemporal scene statis-tics[J]. IEEE Transactions on Image Processing, 2020, 29: 5612-5624.
[14] LE CALLET P, VIARD-GAUDIN C, BARBA D. A convolu-tional neural network approach for objective video quality assessment[J]. IEEE Transactions on Neural Networks, 2006, 17(5): 1316-1327.
[15] KANG L, YE P, LI Y, et al. Convolutional neural networks for no-reference image quality assessment[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 1733-1740.
[16] ZHANG Y, GAO X B, HE L H, et al. Objective video quality assessment combining transfer learning with CNN[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(8): 2716-2730.
[17] KIM W, KIM J, AHN S, et al. Deep video quality assessor: from spatio-temporal visual sensitivity to a convolutional neural aggregation network[C]//LNCS 11205: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 224-241.
[18] VINYALS O, BENGIO S, KUDLUR M. Order matters: se-quence to sequence for sets[J]. arXiv:1511.06391, 2015.
[19] YANG J L, REN P R, ZHANG D Q, et al. Neural aggreg-ation network for video face recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5216-5225.
[20] XU M N, CHEN J M, WANG H Q, et al. C3DVQA: full-reference video quality assessment with 3D convolutional neural network[C]//Proceedings of the 2020 IEEE Inter-national Conference on Acoustics, Speech and Signal Pro-cessing, Barcelona, May 4-8, 2020. Piscataway: IEEE, 2020: 4447-4451.
[21] LI Y D, HE H M, ZHANG Z X. Human motion quality assess-ment toward sophisticated sports scenes based on deeply-learned 3D CNN model[J]. Journal of Visual Communication and Image Representation, 2020, 71: 102702.
[22] LI Y M, PO L M, CHEUNG C H, et al. No-reference video quality assessment with 3D shearlet transform and convo-lutional neural networks[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(6): 1044-1057.
[23] LI Y M, PO L M, XU X Y, et al. No-reference image quality assessment using statistical characterization in the shearlet domain[J]. Signal Processing: Image Communication, 2014, 29(7): 748-759.
[24] WANG C, SU L, HUANG Q. CNN-MR for no reference video quality assessment[C]//Proceedings of the 4th Inter-national Conference on Information Science and Control Engineering, Changsha, Jul 21-23, 2017. Piscataway: IEEE, 2017: 224-228.
[25] AHN S, LEE S. Deep blind video quality assessment based on temporal human perception[C]//Proceedings of the 2018 25th IEEE International Conference on Image Processing, Athens, Oct 7-10, 2018. Piscataway: IEEE, 2018: 619-623.
[26] VARGA D. No-reference video quality assessment based on the temporal pooling of deep features[J]. Neural Processing Letters, 2019, 50: 2595-2608.
[27] ZHANG Y, GAO X B, He L H, et al. Blind video quality assessment with weakly supervised learning and resampling strategy[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(8): 2244-2255.
[28] UTKE M, ZADTOOTAGHAJ S, SCHMIDT S, et al. NDNet-Gaming-development of a no-reference deep CNN for gaming video quality prediction[J]. Multimedia Tools and Appli-cations, 2020: 1-23.
[29] Netflix. VMAF-video multi-method assessment fusion[EB/OL]. (2018-10-02) [2020-09-01]. https://github.com/netflix/vmaf.
[30] RASSOOL R. VMAF reproducibility: validating a perceptual practical video quality metric[C]//Proceedings of the 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, Cagliari, Jun 7-9, 2017. Piscat-away: IEEE, 2017: 1-2.
[31] SHEIKH H R, BOVIK A C. Image information and visual quality[J]. IEEE Transactions on Image Processing, 2006, 15(2): 430-444.
[32] LI S N, ZHANG F, MA L, et al. Image quality assessment by separately evaluating detail losses and additive impair-ments[J]. IEEE Transactions on Multimedia, 2011, 13(5): 935-949.
[33] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recog-nition, Honolulu, Jul 21-26, 2017. Washington: IEEE Com-puter Society, 2017: 2261-2269.
[34] VARGA D, SZIRáNYI T. No-reference video quality assess-ment via pretrained CNN and LSTM networks[J]. Signal, Image and Video Processing, 2019, 13(8): 1569-1576.
[35] HOSU V, HAHN F, JENADELEH M, et al. The Konstanz natural video database (KoNViD-1k)[C]//Proceedings of the 9th International Conference on Quality of Multimedia Experi-ence, Erfurt, May 31-Jun 2, 2017. Piscataway: IEEE, 2017: 1-6.
[36] LI D Q, JIANG T T, JIANG M. Quality assessment of in-the-wild videos[C]//Proceedings of the 27th ACM International Conference on Multimedia, Nice, Oct 21-25, 2019. New York: ACM, 2019: 2351-2359.
[37] TRAN D, BOURDEV L D, FERGUS R, et al. Learning spatio-temporal features with 3D convolutional networks[C]//Pro-ceedings of the 2015 IEEE International Conference on Com-puter Vision, Santiago, Dec 7-13, 2015. Washington: IEEE Computer Society, 2015: 4489-4497.
[38] LIU W T, DUANMU Z F, WANG Z. End-to-end blind quality assessment of compressed videos using deep neural networks[C]//Proceedings?of?the 2018 ACM Multimedia Conference,Seoul, Oct 22-26, 2018. New York: ACM, 2018: 546-554.
[39] MA K D, LIU W T, ZHANG K, et al. End-to-end blind image quality assessment using deep neural networks[J]. IEEE Trans-actions on Image Processing, 2018, 27(3): 1202-1213.
[40] HOU R, ZHAO Y H, HU Y, et al. No-reference video quality evaluation by a deep transfer CNN architecture[J]. Signal Processing: Image Communication, 2020, 83: 115782.
[41] YANG J C, ZHU Y H, MA C F, et al. Stereoscopic video quality assessment based on 3D convolutional neural net-works[J]. Neurocomputing, 2018, 309: 83-93.
[42] YANG J C, LIU T L, JIANG B, et al. 3D panoramic virtual reality video quality assessment based on 3D convolutional neural networks[J].?IEEE Access, 2018, 6: 38669-38682.
[43] WU P, DING W X, YOU Z X, et al. Virtual reality video quality assessment based on 3d convolutional neural net-works[C]//Proceedings of the 2019 IEEE International Con-ference on Image Processing, Taipei, China, Sep 22-25, 2019. Piscataway: IEEE, 2019: 3187-3191.
[44] SESHADRINATHAN K, SOUNDARARAJAN R, BOVIK A C, et al. Study of subjective and objective quality assess-ment of video[J]. IEEE Transactions on Image Processing, 2010, 19(6): 1427-1441.
[45] VU P V, CHANDLER D M. ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices[J]. Journal of Electronic Imaging, 2014, 23(1): 013016.
[46] ZHANG F, LI S N, MA L, et al. IVP subjective quality video database[EB/OL]. [2020-09-01]. http://ivp.ee.cuhk.edu.hk/research/database/subjective/.
[47] CORBILLON X, DE SIMONE F, SIMON G. 360-degree video head movement dataset[C]//Proceedings of the 8th ACM on Multimedia Systems Conference, Taipei, China, Jun 20-23, 2017. New York: ACM, 2017: 199-204.
[48] WU C L, TAN Z H, WANG Z, et al. A dataset for exploring user behaviors in VR spherical video streaming[C]//Proceed-ings of the 8th ACM on Multimedia Systems Conference, Taipei, China, Jun 20-23, 2017. New York: ACM, 2017: 193-198.
[49] DAVID E J, GUTIéRREZ J, COUTROT A, et al. A dataset of head and eye movements for 360°videos[C]//Proceedings of the 9th ACM Multimedia Systems Conference, Amsterdam, Jun 12-15, 2018. New York: ACM, 2018: 432-437.
[50] CHANDLER D M, HEMAMI S S. VSNR: a wavelet-based visual signal-to-noise ratio for natural images[J]. IEEE Transactions on Image Processing, 2007, 16(9): 2284-2298.
[51] MASRY M A, HEMAMI S S. A metric for continuous quality evaluation of compressed video with severe distortions[J]. Signal Processing: Image Communication, 2004, 19(2): 133-146.
[52] URVOY M, BARKOWSKY M, COUSSEAU R, et al. NAMA3DS1-COSPAD1: subjective video quality assessment database on coding conditions introducing freely available high quality 3D stereoscopic sequences[C]//Proceedings of the 4th International Workshop on Quality of Multimedia Experience, Yarra Valley, Jul 5-7, 2012. Piscataway: IEEE,2012: 109-114 .
[53] LIN J Y, SONG R, WU C H, et al. MCL-V: a streaming video quality assessment database[J]. Journal of Visual Com-munication and Image Representation, 2015, 30: 1-9.
[54] LI C, XU M, DU X Z, et al. Bridge the gap between VQA and human behavior on omnidirectional video: a large-scale dataset and a deep learning model[C]//Proceedings of?the?2018 ACM Multimedia Conference, Seoul, Oct 22, 2018. New York: ACM, 2018: 932-940.
[55] LIU L X, WANG T S, HUANG H, et al. Video quality assess-ment using space-time slice mappings[J]. Signal Processing: Image Communication, 2020, 82: 115749.
[56] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[57] SHEIKH H R, BOVIK A C, DE VECIANA G. An infor-mation fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Pro-cessing, 2005, 14(12): 2117-2128.
[58] NGO C W, PONG T C, ZHANG H J. Motion analysis and segmentation through spatio-temporal slices processing[J]. IEEE Transactions on Image Processing, 2003, 12(3): 341-355.
[59] FANG Y M, YAN J B, LI L D, et al. No reference quality assessment for screen content images with both local and global feature representation[J]. IEEE Transactions on Image Processing, 2018, 27(4): 1600-1610.
[60] MA J P, WU J J, LI L D, et al. Active inference of GAN for no-reference image quality assessment[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo, London, Jul 6-10, 2020. Piscataway: IEEE, 2020: 1-6.
[61] BONGINI P, CHIARO R D, BAGDANOV A D, et al. GADA: generative adversarial data augmentation for image quality assessment[C]//LNCS 11752: Proceedings of the 20th Inter-national Conference on Image Analysis and Processing,Trento, Sep 9-13, 2019. Berlin, Heidelberg: Springer, 2019: 214-224.
[62] FANG Y M, WANG J B,WANG J H. No reference quality assessment for stereoscopic images by statistical features [C]//Proceedings of the 9th International Conference on Quality of Multimedia Experience, Erfurt, May 31-Jun 2, 2017. Piscat-away: IEEE, 2017: 1-6.
[63] MITTAL A, MOORTHY A K, BOVIK A C. No-reference im-age quality assessment in the spatial domain[J]. IEEE Trans-actions on Image Processing, 2012, 21(12): 4695-4708.
[64] GAO X B, GAO F, TAO D C, et al. Universal blind image quality assessment metrics via natural scene statistics and multiple kernel learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 24(12): 2013-2026.
[65] YE P, KUMAR J, DOERMANN D S. Beyond human opinion scores: blind image quality assessment based on synthetic scores[C]//Proceedings of the 2014 IEEE Conference on Com-puter Vision and Pattern Recognition, Columbus, Jun 23-28, 2014. Washington: IEEE Computer Society, 2014: 4241-4248.
[66] WU Q B, LI H L, MENG F M, et al. Blind image quality assessment based on multichannel feature fusion and label transfer[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(3): 425-440.
[67] YANG L, WANG H, WEI M. No-reference image quality assessment based on machine learning[J]. Computer Engi-neering and Applications, 2018, 54(19): 34-42.
杨璐, 王辉, 魏敏. 基于机器学习的无参考图像质量评价综述[J]. 计算机工程与应用, 2018, 54(19): 34-42.
[68] ZHANG Y, JIN W Q. Assessment method of fusion image quality in wavelet domain structural similarity[J]. Cinese Journal of Lasers, 2012, 39(S1): 109007.
[69] KIM J, LEE S. Deep learning of human visual sensitivity in image quality assessment framework[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 1969-1977.
[70] ZHANG W X, MA K D, YAN J, et al. Blind image quality assessment using a deep bilinear convolutional neural net-work[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(1): 36-47.
[71] CHETOUANI A. Image quality assessment without reference by mixing deep learning-based features[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo, London, Jul 6-10, 2020. Piscataway: IEEE, 2020: 1-6.
[72] KIM H G, LIM H, RO Y M. Deep virtual reality image quality assessment with human perception guider for omnidirec-tional image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(4): 917-928.
[73] LIM H, KIM H G, RA Y M. VR IQA NET: deep virtual reality image quality assessment using adversarial learning[C]//Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, Apr 15-20, 2018. Piscataway: IEEE, 2018: 6737-6741.
[74] QI F, ZHAO D B, FAN X P, et al. Stereoscopic video quality assessment based on visual attention and just-noticeable dif-ference models[J]. Signal Image & Video Processing, 2016, 10(4): 737-744.
[75] DING X D, LI S M, MA R Z, et al. Objective assessment method for stereo video quality[J]. Acta Scientiarum Natura-lium Universitatis Nankaiensis, 2018, 51(5): 37-43.
丁学东, 李素梅, 马瑞泽, 等. 立体视频质量客观评价方法研究[J]. 南开大学学报(自然科学版), 2018, 51(5): 37-43.
[76] ORDUNA M, DíAZ C, MU?OZ L, et al. Video multimethod assessment fusion (VMAF) on 360VR contents[J]. IEEE Transactions on Consumer Electronics, 2020, 66(1): 22-31.
[77] LI C, XU M, JIANG L, et al. Viewport proposal CNN for 360° video quality assessment[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recog-nition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 10177-10186.
[78] ORDUNA M, PéREZ P, DíAZ C, et al. Evaluating the influence of the HMD, usability, and fatigue in 360VR video quality assessments[C]//Proceedings of the 2020 IEEE Con-ference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, Atlanta, Mar 22-26, 2020. Piscataway: IEEE, 2020: 683-684.
[79] XU M, LI C, ZHANG S Y, et al. State-of-the-art in 360° video/image processing: perception, assessment and compression[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(1): 5-26.
[80] KIM J, KIM W, AHN S, et al. Virtual reality sickness pre-dictor: analysis of visual-vestibular conflict and VR contents[C]//Proceedings of the 2018 10th International Conference on Quality of Multimedia Experience, Cagliari, May 29-Jun 1, 2018. Piscataway: IEEE, 2018: 1-6.
[81] MIN X K, ZHAI G T, ZHOU J T, et al. Study of subjective and objective quality assessment of audio-visual signals[J]. IEEE Transactions on Image Processing, 2020, 29: 6054-6068.
[82] BAMPIS C G, LI Z, BOVIK A C. Spatiotemporal feature integration and model fusion for full reference video quality assessment[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(8): 2256-2270.