Review of Retargeting Methods and Assessment of Retargeted Image Objective Quality

doi:10.3778/j.issn.1673-9418.2404047

Abstract

Abstract: The image retargeting technique enhances compatibility across various display devices by adjusting the size and aspect ratio of the generated image. Though current image retargeting methods have achieved significant research advancements, they are still not applicable to all types of display devices. Artificial distortions may arise in image retargeting, reducing the user􀆳s visual experience and necessitating diverse quality evaluation methods to precisely assess the quality of the resulting retargeted image under various distortion types. This paper comprehensively summarizes the current research progress in image retargeting methods and objective quality evaluation techniques. Firstly, this paper provides an overview of retargeting methods, categorizes them into content-aware image retargeting methods and those based on deep learning techniques, and analyzes the advantages and disadvantages of the two types of methods. Subsequently, this paper delineates the attributes of objective quality evaluation methods for image retargeting, which is essential for the optimization and development of image retargeting methods, encompassing approaches grounded in underlying features and those reliant on multi-level features. This paper then delves into detailing the existing three datasets and conducts a comparative analysis of diverse methodologies for the objective quality evaluation of image retargeting. Finally, this paper proposes potential avenues for future research by addressing the current challenges in the image retargeting field.

Key words: image retargeting, quality assessment, deep learning, multi-level feature

摘要： 图像重定向方法通过改变图像的尺寸与宽高比，使生成的图像在不同的显示设备上均有良好的兼容性。当前已有的图像重定向方法虽然已经取得了一定的研究成果，但它们仍不能适用于所有类型的显示设备。在图像重定向过程中易产生人工失真，降低了用户的视觉体验，针对不同的失真类型，需要使用不同的质量评价方法来准确预测重定向结果图像的质量。对目前图像重定向方法及客观质量评价方法的研究进展均进行了较为全面的总结。总结了重定向方法，分为传统的基于内容感知的图像重定向方法和基于深度学习的图像重定向方法，并分析了这两类方法的优缺点；重点阐述了图像重定向客观质量评价方法的特点，包括基于底层特征的方法和基于多层次特征的方法，质量评价方法对图像重定向方法的优化与发展至关重要；列举了现有的三个数据集，并对不同的图像重定向客观质量评价方法的性能进行了分析比较；根据现阶段图像重定向领域存在的问题，对该方向的未来发展趋势进行了展望。

关键词: 图像重定向, 质量评价方法, 深度学习, 多层次特征

MA Qian, DONG Wu, ZENG Qingtao, ZHANG Yan, LU Likun, ZHOU Ziyi. Review of Retargeting Methods and Assessment of Retargeted Image Objective Quality[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(2): 316-333.

马倩, 董武, 曾庆涛, 张艳, 陆利坤, 周子镱. 图像重定向及客观质量评价方法综述[J]. 计算机科学与探索, 2025, 19(2): 316-333.

References

[1] AVIDAN S, SHAMIR A. Seam carving for content-aware image resizing[J]. ACM Transactions on Graphics, 2007, 26(3): 10.
[2] AVIDAN S, SHAMIR A. Seam carving for content-aware image resizing[C]//Proceedings of the ACM SIGGRAPH 2007. New York: ACM, 2007: 609-617.
[3] NOH H, HAN B, NOH H, et al. Seam carving with forward gradient difference maps[C]//Proceedings of the 20th ACM International Conference on Multimedia. New York: ACM, 2012: 709-712.
[4] AGHCHEHKOHAL M G, KUMARA W G C W. Improved seam carving using meta-heuristics algorithms combination[C]//Proceedings of the 2015 Signal Processing and Intelligent Systems Conference. Piscataway: IEEE, 2015: 43-47.
[5] PENG Y F, LIU X X, WANG G, et al. Irregular shape image retargeting based on seam carving[C]//Proceedings of the 2023 IEEE 3rd International Conference on Electronic Technology, Communication and Information. Piscataway: IEEE, 2023: 1845-1848.
[6] YAN B, LI K, YANG X C, et al. Seam searching-based pixel fusion for image re-targeting[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(1): 15-23.
[7] 郭迎春, 梁云鹤, 于明, 等. 基于图像分块和优化累积能量图的线裁剪算法[J]. 电子与信息学报, 2018, 40(2): 331-337.
GUO Y C, LIANG Y H, YU M, et al. An improved seam carving algorithm based on image blocking and optimized cumulative energy map[J]. Journal of Electronics & Information Technology, 2018, 40(2): 331-337.
[8] HEO Y, KOH Y J, KIM C S. Context-aware seam restoration for image extension[C]//Proceedings of the 2023 IEEE International Conference on Visual Communications and Image Processing. Piscataway: IEEE, 2023: 1-5.
[9] NISHIYAMA M, OKABE T, SATO Y, et al. Sensation-based photo cropping[C]//Proceedings of the 17th ACM International Conference on Multimedia. New York: ACM, 2009: 669-672.
[10] PRITCH Y, KAV-VENAKI E, PELEG S. Shift-map image editing[C]//Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway: IEEE, 2009: 151-158.
[11] SHI M L, YANG L, PENG G Q, et al. A content-aware image resizing method with prominent object size adjusted[C]//Proceedings of the 17th ACM Symposium on Virtual Reality Software and Technology. New York: ACM, 2010: 175-176.
[12] ZHANG L X, SONG H Z, OU Z M, et al. Image retargeting with multifocus fisheye transformation[J]. The Visual Computer, 2013, 29(5): 407-420.
[13] REN T W, LIU Y, WU G S. Image retargeting based on global energy optimization[C]//Proceedings of the 2009 IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2009: 406-409.
[14] WANG Y S, TAI C L, SORKINE O, et al. Optimized scale-and-stretch for image resizing[J]. ACM Transactions on Graphics, 2008, 27(5): 118.
[15] JIN Y, LIU L G, WU Q B. Nonhomogeneous scaling optimization for realtime image resizing[J]. The Visual Computer, 2010, 26(6): 769-778.
[16] GUO Y W, LIU F, SHI J, et al. Image retargeting using mesh parametrization[J]. IEEE Transactions on Multimedia, 2009, 11(5): 856-867.
[17] TAN W M, YAN B, LI K, et al. Image retargeting for preserving robust local feature: application to mobile visual search[J]. IEEE Transactions on Multimedia, 2016, 18(1): 128-137.
[18] KIM Y, EUN H, JUNG C, et al. A quad edge-based grid encoding model for content-aware image retargeting[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(12): 3202-3215.
[19] 陈美颖, 毕秀丽, 刘波. 基于网格与超像素的图像重定向方法[J]. 计算机科学, 2023, 50(S2): 306-313.
CHEN M Y, BI X L, LIU B. Image retargeting method based on grids and superpixels[J]. Computer Science, 2023, 50(S2): 306-313.
[20] RUBINSTEIN M, GUTIERREZ D, SORKINE O, et al. A comparative study of image retargeting[J]. ACM Transactions on Graphics, 2010, 29(6): 160.
[21] WU L F, YAN C C, JIAN M, et al. A fast hybrid retargeting scheme with seam context and content aware strip partition[J]. Neurocomputing, 2018, 286: 198-213.
[22] WANG J Q, QU Z, CHEN Y Y, et al. Adaptive content condensation based on grid optimization for thumbnail image generation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 26(11): 2079-2092.
[23] ZHOU B, WANG X Y, CAO S X, et al. Optimal bi-directional seam carving for compressibility-aware image retargeting[J]. Journal of Visual Communication and Image Representation, 2016, 41: 21-30.
[24] SHAFIEYAN F, KARIMI N, MIRMAHBOUB B, et al. Image seam carving using depth assisted saliency map[C]//Proceedings of the 2014 IEEE International Conference on Image Processing. Piscataway: IEEE, 2014: 1155-1159.
[25] TANG Z H, YAO J M, ZHANG Q, et al. Multi-operator image retargeting with visual quality preservation of salient regions[J]. Multimedia Systems, 2023, 29(2): 811-829.
[26] 彭晏飞, 王静, 刘晓轩, 等. 结合主成分分析和图像分块的重定向研究[J]. 液晶与显示, 2024, 39(2): 157-167.
PENG Y F, WANG J, LIU X X, et al. Retargeting method based on principal component analysis and image blocking[J]. Chinese Journal of Liquid Crystals and Displays, 2024, 39(2): 157-167.
[27] TAN W M, YAN B, LIN C M, et al. Cycle-IR: deep cyclic image retargeting[J]. IEEE Transactions on Multimedia, 2020, 22(7): 1730-1743.
[28] CHO D, PARK J, OH T H, et al. Weakly- and self-supervised learning for content-aware deep image retargeting[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 4568-4577.
[29] LIN J X, ZHOU T K, CHEN Z B. DeepIR: a deep semantics driven framework for image retargeting[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo Workshops. Piscataway: IEEE, 2019: 54-59.
[30] DANON D, ARAR M, COHEN-OR D, et al. Image resizing by reconstruction from deep features[J]. Computational Visual Media, 2021, 7(4): 453-466.
[31] WANG R, HUANG N S, TANG F, et al. Language-driven diversified image retargeting[C]//Proceedings of the SIGGRAPH Asia 2022 Posters. New York: ACM, 2022: 1-2.
[32] NADERI M R, GIVKASHI M H, KARIMI N, et al. OAIR: object-aware image retargeting using PSO and aesthetic quality assessment[EB/OL]. [2024-03-15]. https://arxiv.org/abs/2209.04804.
[33] NADERI M R, GIVKASHI M H, KARIMI N, et al. Aesthetic-aware image retargeting based on foreground-background separation and PSO optimization[J]. Multimedia Tools and Applications, 2024, 83(12): 34867-34886.
[34] GIVKASHI M, NADERI M, KARIMI N, et al. Supervised deep learning for content-aware image retargeting with Fourier convolutions[J]. Multimedia Tools and Applications, 2024, 83(36): 83611-83627.
[35] LIU C Y, NG M K. A one-step image retargeing algorithm based on conformal energy[EB/OL]. [2024-03-15]. https://arxiv.org/abs/2402.18074.
[36] SHOCHER A, BAGON S, ISOLA P, et al. InGAN: capturing and retargeting the “DNA” of a natural image[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 4491-4500.
[37] MASTAN I D, RAMAN S. DCIL: deep contextual internal learning for image restoration and image retargeting[C]//Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2020: 2355-2364.
[38] GRANOT N, FEINSTEIN B, SHOCHER A, et al. Drop the GAN: in defense of patches nearest neighbors as single image generative models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 13450-13459.
[39] LIU S, WEI Z, SUN Y, et al. Composing semantic collage for image retargeting[J]. IEEE Transactions on Image Processing, 2018, 27(10): 5032-5043.
[40] AHMADI M, KARIMI N, SAMAVI S. Context-aware saliency detection for image retargeting using convolutional neural networks[J]. Multimedia Tools and Applications, 2021, 80(8): 11917-11941.
[41] GUO G J, WANG H Z, SHEN C H, et al. Automatic image cropping for visual aesthetic enhancement using deep neural networks and cascaded regression[J]. IEEE Transactions on Multimedia, 2018, 20(8): 2073-2085.
[42] YAN B, NIU X J, BARE B, et al. Semantic segmentation guided pixel fusion for image retargeting[J]. IEEE Transactions on Multimedia, 2019, 22(3): 676-687.
[43] TANG F, DONG W M, MENG Y P, et al. Image retargetability[J]. IEEE Transactions on Multimedia, 2019, 22(3): 641-654.
[44] VALDEZ-BALDERAS D, MURAVEYNYK O, SMITH T. Fast hybrid image retargeting[C]//Proceedings of the 2021 IEEE International Conference on Image Processing. Piscataway: IEEE, 2021: 1849-1853.
[45] GUO Y C, ZHANG M, HAO X K, et al. IRNet-RS: image retargeting network via relative saliency[J]. Neural Computing and Applications, 2024, 36(8): 4133-4149.
[46] WANG Z P, ZHANG W, ZHOU H T. Perception-guided multi-channel visual feature fusion for image retargeting[J]. Signal Processing: Image Communication, 2019, 79: 63-70.
[47] ZHOU Y Z, ZHANG L M, ZHANG C, et al. Perceptually aware image retargeting for mobile devices[J]. IEEE Transactions on Image Processing, 2018, 27(5): 2301-2313.
[48] 鄢杰斌, 方玉明, 刘学林. 图像质量评价研究综述: 从失真的角度[J]. 中国图象图形学报, 2022, 27(5): 1430-1466.
YAN J B, FANG Y M, LIU X L. The review of distortion-related image quality assessment[J]. Journal of Image and Graphics, 2022, 27(5): 1430-1466.
[49] 陈慧, 李朝锋. 深度卷积神经网络的立体彩色图像质量评价[J]. 计算机科学与探索, 2018, 12(8): 1315-1322.
CHEN H, LI C F. Stereoscopic color image quality assessment via deep convolutional neural network[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(8): 1315-1322.
[50] 曹玉东, 刘海燕, 贾旭, 等. 基于深度学习的图像质量评价方法综述[J]. 计算机工程与应用, 2021, 57(23): 27-36.
CAO Y D, LIU H Y, JIA X, et al. Overview of image quality assessment method based on deep learning[J]. Computer Engineering and Applications, 2021, 57(23): 27-36.
[51] HSU C C, LIN C W, FANG Y M, et al. Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss[J]. IEEE Journal of Selected Topics in Signal Processing, 2014, 8(3): 377-389.
[52] LIN J X, ZHU L L, CHEN Z B, et al. Objective quality assessment for image retargeting based on hybrid distortion pooled model[C]//Proceedings of the 2015 7th International Workshop on Quality of Multimedia Experience. Piscataway: IEEE, 2015: 1-6.
[53] JIANG Q P, SHAO F, LIN W S, et al. Learning sparse representation for objective image retargeting quality assessment[J]. IEEE Transactions on Cybernetics, 2018, 48(4): 1276-1289.
[54] KARIMI M, SAMAVI S, KARIMI N, et al. Quality assessment of retargeted images by salient region deformity analysis[J]. Journal of Visual Communication and Image Representation, 2017, 43: 108-118.
[55] KARIMI M, SAMAVI S, KARIMI N, et al. Pyramidal modeling of geometric distortions for retargeted image quality evaluation[J]. Multimedia Tools and Applications, 2018, 77(11): 13799-13820.
[56] ZHANG Y B, LIN W S, LI Q H, et al. Multiple-level feature-based measure for retargeted image quality[J]. IEEE Transactions on Image Processing, 2018, 27(1): 451-463.
[57] ABSETAN A, FATHI A. Integration of deep learned and handcrafted features for image retargeting quality assessment[J]. Cybernetics and Systems, 2023, 54(5): 673-696.
[58] LI L D, LI Y X, WU J J, et al. Quality evaluation for image retargeting with instance semantics[J]. IEEE Transactions on Multimedia, 2020, 23: 2757-2769.
[59] FANG Y M, ZENG K, WANG Z, et al. Objective quality assessment for image retargeting based on structural similarity[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2014, 4(1): 95-105.
[60] ZHANG T T, YU M, GUO Y C, et al. Content-aware retargeted image quality assessment[J]. Information, 2019, 10(3): 111.
[61] ZHOU B, LIU Z A, JI J Y, et al. Objective quality assessment of retargeted images based on RBF neural network with structural distortion and content change[J]. Multimedia Tools and Applications, 2023, 82(5): 7463-7477.
[62] PENG Z Y, JIANG Q P, SHAO F, et al. LGGD: image retargeting quality assessment by measuring local and global geometric distortions[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(6): 3422-3437.
[63] WEI L S, ZHAO L, PENG J. Reduced reference quality assessment for image retargeting by earth mover􀆳s distance[J]. Applied Sciences, 2021, 11(20): 9776.
[64] JIANG B, YANG J C, MENG Q G, et al. A deep evaluator for image retargeting quality by geometrical and contextual interaction[J]. IEEE Transactions on Cybernetics, 2018, 50(1): 87-99.
[65] TSAO C, SU P C. Quality assessment of image retargeting based on importance of objects[C]//Proceedings of the 2023 International Conference on Consumer Electronics. Piscataway: IEEE, 2023: 779-780.
[66] 陈楠, 蔡柏亲, 杨恭宁, 等. 考虑结构一致性和内容完整性的重定向图像质量评价[J]. 福州大学学报(自然科学版), 2023, 51(4): 517-524.
CHEN N, CAI B Q, YANG G N, et al. Image retargeting quality assessment using content integrity and structural consistency[J]. Journal of Fuzhou University (Natural Science Edition), 2023, 51(4): 517-524.
[67] WU J N, XIE R, SONG L, et al. Deep feature guided image retargeting[C]//Proceedings of the 2019 IEEE Conference on Visual Communications and Image Processing. Piscataway: IEEE, 2019: 1-4.
[68] TANG Z H, YAO J M. Image retargeting quality assessment based on saliency-driven classification[J]. Journal of Signal Processing: Image Communication, 2022, 105: 116688.
[69] LIANG Y, LIU Y J, GUTIERREZ D. Objective quality prediction of image retargeting algorithms[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(2): 1099-1110.
[70] YAN B, BARE B, LI K, et al. Learning quality assessment of retargeted images[J]. Signal Processing: Image Communication, 2017, 56: 12-19.
[71] LIU Y J, HAN Y H, YE Z P, et al. Ranking-preserving cross-source learning for image retargeting quality assessment[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(7): 1798-1805.
[72] NIU Y Z, ZHANG S, WU Z S, et al. Image retargeting quality assessment based on registration confidence measure and noticeability-based pooling[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(3): 972-985.
[73] RUBINSTEIN M, SHAMIR A, AVIDAN S. Multi-operator media retargeting[J]. ACM Transactions on Graphics, 2009, 28(3): 1-11.
[74] MA L, LIN W S, DENG C W, et al. Image retargeting quality assessment: a study of subjective scores and objective metrics[J]. IEEE Journal of Selected Topics in Signal Processing, 2012, 6(6): 626-639.
[75] 桑庆兵, 程大宇. 稀疏表示的无参考图像质量评价方法[J]. 计算机科学与探索, 2017, 11(1): 144-154.
SANG Q B, CHENG D Y. Blind image quality assessment via sparse representation[J]. Journal of Frontiers of Computer Science and Technology, 2017, 11(1): 144-154.
[76] 方玉明, 眭相杰, 鄢杰斌, 等. 无参考图像质量评价研究进展[J]. 中国图象图形学报, 2021, 26(2): 265-286.
FANG Y M, SUI X J, YAN J B, et al. Progress in no-reference image quality assessment[J]. Journal of Image and Graphics, 2021, 26(2): 265-286.
[77] 艾达, 白岩松, 于可欣, 等. 全景图像质量评价方法最新进展[J]. 计算机工程与应用, 2022, 58(24): 1-11.
AI D, BAI Y S, YU K X, et al. Recent advances on panoramic image quality assessment methods[J]. Computer Engineering and Applications, 2022, 58(24): 1-11.