Aesthetic Feature Image Generation Method Embedded with Self-Attention Mechanism

doi:10.3778/j.issn.1673-9418.2006075

Abstract

Abstract:

In order to solve the problems such as the single structure of the generated image, the lack of rich detail features, resulting in the lack of aesthetic sense, this paper proposes an aesthetic feature image generation method embedded with self-attention mechanism. In order to increase the aesthetic characteristics of the generated image, the correlation between the image aesthetic evaluation criteria and the generation model is studied, and the aesthetic loss function based on aesthetic score is defined. In order to ensure the consistency of the generated image and the real image in semantic content, VGG network is added to construct the content loss function, the Charbonnier loss is used to replace the L1 loss, and the aesthetic loss, content loss and the confrontation loss of the evolutionary generation confrontation network are combined in a weighted form, to guide and optimize image generation. The self-attention mechanism module is introduced into the generator and discriminator, and the dense blocks are added before the self-attention mechanism module of the generator to fully extract the features, which is conducive to the self-attention mechanism to obtain more internal global dependencies of the features, and promoting the generated image to have clear details and rich texture features. The experimental results on Cifar10 and CUHKPQ data sets show that the method is effective in improving the image aesthetic effect. The Fréchet distance value is increased by 3.21 and 5.44 respectively compared with that of the evolutionary generation confrontation network, and the image aesthetic score value is improved by 0.75 and 0.88 compared with the evolutionary generation confrontation network.

Key words: evolutionary generative confrontation network, image aesthetics, self-attention, loss function, image generation

摘要：

针对生成的图像结构单一，细节特征不够丰富，导致美观感不足等问题，提出了一种嵌入自注意力机制的美学特征图像生成方法。为了增加生成图像的美学特征，研究图像美学评价标准与生成模型之间的关联性，定义了基于美学分数的美学损失函数；为保证生成图像与真实图像在语义内容上的一致性，加入VGG网络，构造内容损失函数，采用Charbonnier损失代替L1损失，并将美学损失、内容损失和进化生成对抗网络的对抗损失以加权形式组合，引导与优化图像的生成。在生成器和判别器中引入自注意力机制模块，并将密集卷积块加入生成器自注意力机制模块之前，充分提取特征，有利于自注意力机制高效获取更多特征内部的全局依赖关系，促使生成图像细节清晰，纹理特征丰富。在Cifar10、CUHKPQ两个数据集上的实验结果表明该方法在提升图像美学效果方面是有效的，其弗雷歇距离值相较于进化生成对抗网络分别提高了3.21和5.44，图像美学分数值相较于进化生成对抗网络分别提高了0.75和0.88。

关键词: 进化生成对抗网络, 图像美学, 自注意力, 损失函数, 图像生成

MA Li, ZOU Yali. Aesthetic Feature Image Generation Method Embedded with Self-Attention Mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(9): 1728-1739.

马力, 邹亚莉. 嵌入自注意力机制的美学特征图像生成方法[J]. 计算机科学与探索, 2021, 15(9): 1728-1739.

References

[1] FAZI M B, FULLER M. Computational aesthetics[M]. New York: John Wiley & Sons, Inc., 2016.
[2] HOENIG F. Defining computational aesthetics[C]//Proceed-ings of the 1st Eurographics Workshop on Computational Aesthetics in Graphics, Visualization, and Imaging, Girona, May 18-20, 2005. Eurographics Association, 2005: 13-18.
[3] WANG W N, YI J J, HE Q H. Review for computational image aesthetics[J]. Journal of Image and Graphics, 2012, 17(8): 893-901.
王伟凝, 蚁静缄, 贺前华. 可计算图像美学研究进展[J]. 中国图象图形学报, 2012, 17(8): 893-901.
[4] YANG H T, SHI P, HE S K, et al. A comprehensive survey on image aesthetic quality assessment[C]//Proceedings of the 18th IEEE/ACIS International Conference on Computer and Information Science, Beijing, Jun 17-19, 2019. Piscataway:IEEE, 2019: 294-299.
[5] DOSHI N, SHIKHENAWIS G, MITRA S K. Image aesthetics assessment using multi channel convolutional neural networks[C]//LNCS 1148: Proceedings of the 4th International Con-ference on Computer Vision and Image Processing, Jaipur, Sep 27-29, 2019. Berlin, Heidelberg: Springer, 2019: 15-24.
[6] GE R X. A method of image aesthetics classification based on color harmony and composition[J]. Software Guide, 2017, 16(11): 221-224.
葛瑞雪. 融合色彩和谐性与构图的图像美学分类方法研究[J]. 软件导刊, 2017, 16(11): 221-224.
[7] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[J]. arXiv:1406.2661v1, 2014.
[8] NGUYEN D T, PHAM T D, BATCHULUUN G, et al. Pres-entation attack face image generation based on a deep gen-erative adversarial network[J]. Sensors, 2020, 20(7): 1810.
[9] LIN J X, XIA Y C, QIN T, et al. Conditional image-to-image translation[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 5524-5532.
[10] CHERIAN A, SULLIVAN A. Sem-GAN: semantically-consistent image-to-image translation[C]//Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision, Waikoloa Village, Jan 7-11, 2019. Piscataway: IEEE, 2019: 1797-1806.
[11] CHEN Y Z, HU H F. An improved method for semantic image inpainting with GANs: progressive inpainting[J]. Neural Processing Letters, 2019, 49(3): 1355-1367.
[12] ZHANG N, JI H, LIU L, et al. Exemplar-based image inpaint-ing using angle-aware patch matching[J]. EURASIP Journal on Image and Video Processing, 2019: 70.
[13] EHSANI K, MOTTAGHI R, FARHADI A. SeGAN: seg-menting and generating the invisible[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-22, 2018. Washington:IEEE Computer Society, 2018: 6144-6153.
[14] SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J]. Acta Auto-matica Sinica, 2017, 43(5): 697-709.
孙旭, 李晓光, 李嘉锋, 等. 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43(5): 697-709.
[15] LIM B, SON S, KIM H, et al. Enhanced deep residual net-works for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 1132-1140.
[16] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[J]. arXiv:1701.07875, 2017.
[17] SINN M, RAWAT A. Non-parametric estimation of Jensen-Shannon divergence in generative adversarial network train-ing[C]//Proceedings of the 2018 International Conference on Artificial Intelligence and Statistics, Playa Blanca, Apr 9-11, 2018: 642-651.
[18] MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral no-rmalization for generative adversarial networks[C]//Proce-edings of the 6th International Conference on Learning Re-presentations, Vancouver, Apr 30-May 3, 2018: 1-26.
[19] WANG C Y, XU C, YAO X, et al. Evolutionary generative adversarial networks[J]. IEEE Transactions on Evolutionary Computation, 2019, 23(6): 921-934.
[20] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[J]. arXiv:1805.08318, 2018.
[21] ZHU L Y. Aesthetic dictionary[M]. Shanghai: Shanghai Le-xicographical Publishing House, 2010.
[22] MAI L, JIN H L, LIU F. Composition-preserving deep photo aesthetics assessment[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 497-506.
[23] DENG X, CUI C, FANG H, et al. Personalized image aesth-etics assessment[C]//Proceedings of the 2017 ACM on Con-ference on Information and Knowledge Management. New York: ACM, 2017: 2043-2046.
[24] DATTA R, JOSHI D, LI J, et al. Studying aesthetics in photo-graphic images using a computational approach[C]//LNCS 3953: Proceedings of the 9th European Conference on Com-puter Vision, Graz, May 7-13, 2006. Berlin, Heidelberg: Springer, 2006: 288-301.
[25] DHAR S, ORDONEZ V, BERG T L. High level describable attributes for predicting aesthetics and interestingness[C]// Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, Jun 20-25, 2011. Washington: IEEE Computer Society, 2011: 1657-1664.
[26] KONG S, SHEN X H, LIN Z L, et al. Photo aesthetics ranking network with attributes and content adaptation[C]// LNCS 9905: Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham:Springer, 2016: 662-679.
[27] TALEBI H, MILANFAR P. NIMA: neural image assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011.
[28] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 2261-2269.
[29] SHAO Q, MA H P. Convolutional neural network text class-ification model with self-attention mechanism[J]. Journal of Chinese Computer Systems, 2019, 40(6): 1137-1141.
邵清, 马慧萍. 融合self-attention机制的卷积神经网络文本分类模型[J]. 小型微型计算机系统, 2019, 40(6): 1137-1141.
[30] XU T Y, WANG Z. Text-to-image synthesis optimization based on aesthetic assessment[J]. Journal of Beijing Univer-sity of Aeronautics and Astronautics, 2019, 45(12): 2438-2448.
徐天宇, 王智. 基于美学评判的文本生成图像优化[J]. 北京航空航天大学学报, 2019, 45(12): 2438-2448.
[31] BARRON J T. A general and adaptive robust loss function[J]. arXiv:1701.03077, 2017.