Emotional Rendering of 3D Indoor Scene with Chinese Elements

doi:10.3778/j.issn.1673-9418.2210096

Abstract

Abstract: One of the challenging tasks is to use computer technology to automatically design a virtual indoor scene that both satisfies realness and matches the target emotion. The subjective nature of emotions brings uncertainty of results. At present, there is a lack of approach to identify and evaluate emotion of indoor scenes. In addition, under the premise of fully considering emotional appeals, the authenticity of scene is also one of important factors in indoor scene design. Aiming at above problems, a novel optimization algorithm combining Chinese elements for indoor scenes rendering is proposed. Firstly, an emotion classifier is trained to identify and evaluate the emotion with the features extracted via deep learning from a indoor scene dataset containing 25000 images. Secondly, in order to ensure the authenticity of rendering results, an algorithm is proposed to evaluate how realistic the colors of the objects’ textures. Next, an algorithm is designed to render indoor scene automatically according to the target emotion. Then, a style transfer algorithm integrating with Chinese elements is used to carry out fine-grained refinement processing on the furnishings in an indoor scene, improve the spatial connotation, cultural connotation and emotional expression of rendering results, and enhance the visual appeal. Finally, the approach is tested in four indoor scenes, and the correctness and effectiveness of the approach are verified through statistical analysis of results and user survey data.

Key words: virtual reality, affective modeling;?Chinese elements, style transfer

摘要： 情绪具有主观性，利用计算机技术自动生成一个既满足真实性又与目标情绪匹配的虚拟室内场景是一项充满挑战的任务。目前缺乏对室内场景情感表达进行识别和评估的技术方法，且在充分考虑情感诉求的前提下，如何提升场景渲染结果真实性也是室内场景设计需要考虑的重要因素。针对上述问题，提出了一种融合中式元素的虚拟室内场景情绪化渲染算法。首先，利用深度学习算法从一个包含25 000张图片的家庭室内场景数据集中提取不同情绪的特征，训练情绪分类器，用于识别和评估渲染过程中虚拟室内场景的情绪表达。其次，为了保证渲染结果真实性，设计了一种场景物体纹理颜色真实性度量算法。然后，研究了根据目标情绪自动渲染虚拟室内场景的优化算法、融合中式元素的风格迁移算法，对场景中的物体进行细粒度的中式风格化处理，提升渲染结果的空间内涵、文化底蕴以及情感表达，增强视觉感染力。最后，在四个不同的室内场景中对该算法进行了实验，并通过对实验结果和用户调研数据的统计分析验证了该算法的正确性和有效性。

关键词: 虚拟现实, 情感建模, 中式元素, 风格迁移

SHENG Jiachuan, HU Guolin, LI Yuzhi. Emotional Rendering of 3D Indoor Scene with Chinese Elements[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(2): 465-476.

盛家川, 胡国林, 李玉芝. 融合中式元素的三维室内场景情绪化渲染[J]. 计算机科学与探索, 2024, 18(2): 465-476.

References

[1] SCHOENLEIN M A, CAMPOS J, LANDE K J, et al. Unifying effects of direct and relational associations for visual communication[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 385-395.
[2] 盛家川, 陈雅琦, 王君, 等. 深度学习结构优化的图像情感分类[J]. 红外与激光工程, 2020, 49(11): 264-273.
SHENG J C, CHEN Y Q, WANG J, et al. Image sentiment classification via deep learning structure optimization[J]. Infrared and Laser Engineering, 2020, 49 (11): 264-273.
[3] BRODSCHELM L, GR?BER F, HIEBER D, et al. Mood adaptive display coloring-utilizing modern machine learning techniques and intelligent coloring to influence the mood of PC users[C]//Proceedings of the 14th International Conference on Advances in Computer-Human Interactions, Nice, Jul 18-22, 2021: 48-54.
[4] LIU S, LUO H. Hierarchical emotional color theme extraction[J]. Color Research & Application, 2016, 41(5): 513-522.
[5] LIU D, JIANG Y, PEI M, et al. Emotional image color transfer via deep learning[J]. Pattern Recognition Letters, 2018, 110: 16-22.
[6] LIU S, PEI M. Texture-aware emotional color transfer between images[J]. IEEE Access, 2018, 6: 31375-31386.
[7] LIU S, WANG H, PEI M. Facial-expression-aware emotional color transfer based on convolutional neural network[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2022, 18(1): 1-19.
[8] YAN C, SHAO B, ZHAO H, et al. 3D room layout estimation from a single RGB image[J]. IEEE Transactions on Multimedia, 2020, 22(11): 3014-3024.
[9] JIN T, ZHUANG J, XIAO J, et al. High precision indoor model contour extraction algorithm based on geometric information[C]//Proceedings of the 2021 IEEE 3rd International Conference on Frontiers Technology of Information and Computer, Greenville, Nov 12-14, 2021. Piscataway: IEEE, 2021: 188-193.
[10] WANG Z, PHILION J, FIDLER S, et al. Learning indoor inverse rendering with 3D spatially-varying lighting[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 12538-12547.
[11] ZHANG S H, ZHANG S K, XIE W Y, et al. Fast 3D indoor scene synthesis by learning spatial relation priors of objects[J]. IEEE Transactions on Visualization and Computer Gra-phics, 2021, 28(9): 3082-3092.
[12] SHI Z, SHEN Y, ZHU J, et al. 3D-aware indoor scene synthesis with depth priors[J]. arXiv:2202.08553, 2022.
[13] MA R, PATIL A G, FISHER M, et al. Language-driven synthesis of 3D scenes from scene databases[J]. ACM Transactions on Graphics, 2018, 37(6): 1-16.
[14] GUO S, SHI Y, XIAO P, et al. Creative and progressive interior color design with eye-tracked user preference[J]. ACM Transactions on Computer-Human Interaction, 2023, 30(1): 1-31.
[15] FU H, CAI B, GAO L, et al. 3D-FRONT: 3D furnished rooms with layouts and semantics[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 10933-10942.
[16] LIN J, XIAO P, FU Y, et al. C3 assignment: camera cubemap color assignment for creative interior design[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(8): 2895-2908.
[17] FU Q, YAN H, FU H, et al. Interactive design and preview of colored snapshots of indoor scenes[J]. Computer Graphics Forum, 2020, 39(7): 543-552.
[18] ZHU J, GUO Y, MA H. A data-driven approach for furniture and indoor scene colorization[J]. IEEE Transactions on Visualization and Computer Graphics, 2017, 24(9): 2473-2486.
[19] JEONG J, JO J, CHO S, et al. 3D scene painting via semantic image synthesis[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, Jun 19-24, 2022. Piscataway: IEEE, 2022: 2262-2272.
[20] HE B, GAO F, MA D, et al. ChipGAN: a generative adversarial network for Chinese ink wash painting style transfer[C]//Proceedings of the 26th ACM International Conference on Multimedia, Seoul, Oct 22-26, 2018. New York: ACM, 2018: 1172-1180.
[21] SHENG J C, SONG C F, WANG J, et al. Convolutional neural network style transfer towards Chinese paintings[J]. IEEE Access, 2019, 7: 163719-163728.
[22] 盛家川, 董玙璠, 李小妹, 等. 中国肖像画的风格转移算法[J]. 模式识别与人工智能, 2021, 34(6): 509-521.
SHENG J C, DONG Y F, LI X M, et al. Chinese portrait painting style transfer algorithm[J]. Pattern Recognition and Artificial Intelligence, 2021, 34(6): 509-521.
[23] 盛家川, 陈雅琦, 韩亚洪. 深层网络特征聚合重标定的中国画情感分类算法[J]. 计算机辅助设计与图形学学报, 2020, 32(9): 1420-1429.
SHENG J C, CHEN Y Q, HAN Y H. Sentiment classification of Chinese paintings via feature recalibration of deep network aggregation[J].?Journal of Computer-Aided Design & Computer Graphics, 2020, 32(9): 1420-1429.
[24] SHARMA G, WU W, DALAL E. The CIEDE2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations[J]. Color Research & Application, 2005, 30(1): 21-30.
[25] BELL S, UPCHURCH P, SNAVELY N, et al. OpenSurfaces: a richly annotated catalog of surface appearance[J]. ACM Transactions on Graphics, 2013, 32(4): 1-17.
[26] MOHSENI S A, WU H R, THOM J A, et al. Recognizing induced emotions with only one feature: a novel color histogram-based system[J]. IEEE Access, 2020, 8: 37173-37190.
[27] 王征, 李皓月, 许洪山, 等. 基于卷积神经网络和SVM的中国画情感分类[J]. 南京师大学报(自然科学版), 2017, 40(3): 74-79.
WANG Z, LI H Y, XU H S, et al. Chinese painting emotion classification based on convolution neural network and SVM[J]. Journal of Nanjing Normal University (Natural Science Edition), 2017, 40(3): 74-79.