Single Image Super-Resolution Reconstruction Method for Generative Adversarial Network

doi:10.3778/j.issn.1673-9418.1910067

Abstract

Abstract:

The super-resolution reconstruction method based on deep convolutional neural network has a high peak signal-to-noise ratio (PSNR), but the reconstruction results have the problem of lack of high-frequency information and texture details and poor visual perception under large-scale factors. Aiming at this problem, a single image super-resolution reconstruction method based on generative adversarial network is proposed. Firstly, the hinge loss in the migration support vector machine is taken as the objective function, and then the Charbonnier loss which is more stable and more anti-noise is used instead of the L2 loss function. Finally, the batch normalization layer which is unfavorable to the super resolution of the image in the residual block and discriminator is removed, and the spectral normalization is used in the generator and discriminator to reduce the computational overhead and stabilize the model training. The experimental results of 4X upscaling show that compared with other comparison methods, the PSNR value of the reconstructed image is improved by up to 4.6 dB and the SSIM value is increased by 0.1, and the test time is shorter. The experimental data and effect diagram show that the super-resolution image reconstructed by this method has better visual effect and higher PSNR and SSIM values.

Key words: super-resolution reconstruction, generative adversarial network (GAN), deep learning, convolutional neural network (CNN), loss function

摘要：

基于深度卷积神经网络的超分辨率重建方法虽然有较高的峰值信噪比（PSNR），但重建结果在大尺度因子下存在缺乏高频信息和纹理细节，视觉感知效果差的问题。针对这一问题，提出了一种基于生成对抗网络的单图像超分辨率重建方法。首先迁移支持向量机中的hinge损失作为目标函数，其次使用更加稳定、抗噪性更强的Charbonnier损失代替L2损失函数，最后去掉了残差块和判别器中对图像超分辨率不利的批规范化层，并在生成器和判别器中使用谱归一化来减小计算开销，稳定模型训练。实验结果表明，在4倍放大尺度因子下，相较其他对比方法，该方法重建图像的PSNR值最高提升4.6 dB，SSIM值最高提升0.1，测试时间较短。实验数据和效果图均表明该方法重建的超分辨率图像视觉效果较好，且有更高的PSNR和SSIM值。

关键词: 超分辨率重建, 生成对抗网络（GAN）, 深度学习, 卷积神经网络（CNN）, 损失函数

PENG Yanfei, GAO Yi, DU Tingting, SANG Yu, ZI Lingling. Single Image Super-Resolution Reconstruction Method for Generative Adversarial Network[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(9): 1612-1620.

彭晏飞，高艺，杜婷婷，桑雨，訾玲玲. 生成对抗网络的单图像超分辨率重建方法[J]. 计算机科学与探索, 2020, 14(9): 1612-1620.

References

[1] Li X, Wei H W, Zhang H Q. Super-resolution reconstruc-tion of single remote sensing image combined with deep learning[J]. Journal of Image and Graphics, 2018, 23(2): 209-218.李欣, 韦宏卫, 张洪群. 结合深度学习的单幅遥感图像超分辨率重建[J]. 中国图象图形学报, 2018, 23(2): 209-218.
[2] Zou W W W, Yuen P C. Very low resolution face recogni-tion problem[J]. IEEE Transactions on Image Processing, 2010, 21(1): 1-6.
[3] Lei S, Shi Z W, Zou Z X. Super-resolution for remote sens-ing images via local-global combined network[J]. IEEE Geo-science and Remote Sensing Letters, 2017, 14(8): 1243-1247.
[4] Ren H Y, El-Khamy M, Lee J. CT-SRCNN: cascade trained and trimmed deep convolutional neural networks for image super resolution[C]//Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision, Lake Tahoe, Mar 12-15, 2018. Washington: IEEE Computer Society, 2018: 1423-1431.
[5] Zhu S Y, Zeng B, Liu G H, et al. Image interpolation based on non-local geometric similarities[C]//Proceedings of the 2015 IEEE International Conference on Multimedia and Expo, Turin, Jun 29-Jul 3, 2015: 1-6.
[6] Zhang K B, Gao X B, Tao D C, et al. Single image super-resolution with non-local means and steering kernel regres-sion[J]. IEEE Transactions on Image Processing, 2012, 21(11): 4544-4556.
[7] Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[8] Shi W Z, Caballero J, Huszár F, et al. Real-time single im-age and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recogni-tion, Las Vegas, Jun 27-30, 2016. Washington: IEEE Com-puter Society, 2016: 1874-1883.
[9] Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 1646-1654.
[10] Lim B, Son S, Kim H, et al. Enhanced deep residual net-works for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, Jul 21-26, 2017. Wash-ington: IEEE Computer Society, 2017: 136-144.
[11] Lai W S, Huang J B, Ahuja N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Procee-dings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 5835-5843.
[12] Haris M, Shakhnarovich G, Ukita N. Deep back-projection networks for super-resolution[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recogni-tion, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 1664-1673.
[13] Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Genera-tive adversarial nets[C]//Proceedings of the Annual Confer-ence on Neural Information Processing Systems 2014, Mon-treal, Dec 8-13, 2014. Red Hook: Curran Associates, 2014: 2672-2680.
[14] Ledig C, Theis L, Huszár F, et al. Photo-realistic single im-age super-resolution using a generative adversarial network[C]//Proceedings of the 2017 IEEE Conference on Com-puter Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 105-114.
[15] Wang X T, Yu K, Wu S W, et al. ESRGAN: enhanced super-resolution generative adversarial networks[C]//LNCS 11133: Proceedings of the 2018 European Conference on Com-puter Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 63-79.
[16] Zhang H, Goodfellow I J, Metaxas D N, et al. Self-attention generative adversarial networks[J]. arXiv: 1805.08318, 2018.
[17] Brock A, Donahue J, Simonyan K. Large scale GAN train-ing for high fidelity natural image synthesis[J]. arXiv:1809.11096, 2018.
[18] Miyato T, Kataoka T, Koyama M, et al. Spectral normaliza-tion for generative adversarial networks[J]. arXiv:1802.05957, 2018.
[19] Timofte R, Agustsson E, van Gool L, et al. NTIRE 2017 challenge on single image super-resolution: methods and results[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Hon-olulu, Jul 21-26, 2017. Washington: IEEE Computer Society, 2017: 1110-1121.
[20] Bevilacqua M, Roumy A, Guillemot C, et al. Neighbor embedding based single-image super-resolution using semi-nonnegative matrix factorization[C]//Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Sig-nal Processing, Kyoto, Mar 25-30, 2012. Piscataway: IEEE, 2012: 1289-1292.
[21] Yuan Y, Liu S Y, Zhang J W, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, Jun 18-22, 2018. Washington: IEEE Computer Society, 2018: 701-710.