Image Super-resolution Reconstruction of Functional Composite Deep Network

doi:10.3778/j.issn.1673-9418.1909006

Abstract

Abstract:

Aiming at the problems of weak inter-layer connections, single network concerns and insufficient utilization of layered features in the existing single-image super-resolution reconstruction using simple chain-stacked single network, a composite depth neural network is proposed to improve the performance of image super-resolution reconstruction. Firstly, feature extraction layer is used to extract the initial features of low-resolution images. Then, the initial features are fed into two sub-networks, one is responsible for image detail extraction and calculation, the other is responsible for image noise degradation and elimination. Then the deep abstract features output by the two sub-networks are combined with the initial features, and finally the super-resolution image is obtained through the reconstruction layer. Taking peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indices, the experiment is carried out by using amplification factor 3 in Set14 test set. The experimental results show that PSNR is increased by 2.27 dB, 0.66 dB, 0.54 dB, 0.05 dB and 0.21 dB respectively, while SSIM is improved by 6.08, 1.54, 1.41, 0.36， 0.09 percentage points by comparing the combined network with algorithms Bicubic, SelfEx, SRCNN, VDSR and RED.

Key words: single image super-resolution reconstruction, convolutional neural network (CNN), composite network, sub-network, feature combination

摘要：

针对现有单图像超分辨率重建时主要采用的简单链式堆叠的单一网络存在层间联系弱、网络关注点单一以及分层特征不能充分利用等问题，提出了一种复合的深度神经网络用于提升图像超分辨重建性能。该方法首先使用特征提取层提取低分辨率图像的初始特征；再将初始特征分别送入两个子网络，一个子网络负责图像细节的提取与运算，另一子网络负责图像噪声降解与消除；然后将两个子网络输出的深层次抽象特征与初始特征相结合，最后通过重建层获得超分辨率图像。以峰值信噪比（PSNR）与结构相似性（SSIM）为评价指标，在Set14测试集上使用放大因子3进行实验，将复合网络与算法Bicubic、SelfEx、SRCNN、VDSR和RED等进行对比，实验结果发现，PSNR分别提高了2.27 dB、0.66 dB、0.54 dB、0.05 dB、0.21 dB，而SSIM则分别提高了6.08、1.54、1.41、0.36、0.09个百分点。

关键词: 单图像超分辨率重建, 卷积神经网络（CNN）, 复合网络, 子网络, 特征结合

TANG Jiajun, LIU Hui, HU Xueying. Image Super-resolution Reconstruction of Functional Composite Deep Network[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(8): 1368-1379.

唐家军，刘辉，胡雪影. 功能型复合深度网络的图像超分辨率重建[J]. 计算机科学与探索, 2020, 14(8): 1368-1379.

References

[1] Su H, Zhou J, Zhang Z H. Survey of super-resolution image reconstruction methods[J]. Acta Automatica Sinica, 2013, 39(8): 1202-1213.苏衡, 周杰, 张志浩. 超分辨率图像重建方法综述[J]. 自动化学报, 2013, 39(8): 1202-1213.
[2] Glasner D, Bagon S, Irani M. Super-resolution from a single image[C]//Proceedings of the 12th IEEE International Con-ference on Computer Vision, Kyoto, Sep 27-Oct 4, 2009. Was-hington: IEEE Computer Society, 2009: 349-356.
[3] Park S C, Park M K, Kang M G. Super-resolution image rec-onstruction: a technical overview[J]. IEEE Signal Proce-ssing Magazine, 2003, 20(3): 21-36.
[4] Wang H P, Zhou L L, Zhang J. Region-based bicubic image interpolation algorithm[J]. Computer Engineering, 2010, 36(19): 216-218.王会鹏, 周利莉, 张杰. 一种基于区域的双三次图像插值算法[J]. 计算机工程 , 2010, 36(19): 216-218.
[5] Yang J C, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873.
[6] Timofte R, De Smet V, Van Gool L. A+: adjusted anchored neighborhood regression for fast super-resolution[C]//LNCS 9006: Proceedings of the 12th Asian Conference on Computer Vision, Singapore, Nov 1-5, 2014. Berlin, Heidelberg: Springer, 2015: 111-126.
[7] Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recog-nition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 5197-5206.
[8] Hayat K. Multimedia super-resolution via deep learning: a survey[J]. Digital Signal Processing, 2018, 81(10): 198-217.
[9] Tan Z, Xiang L B, Lü Q B, et al. A sequence images super-resolution enhancement approach based on frequency-domain[J]. Acta Optica Sinica, 2017, 37(7): 83-88.谭政, 相里斌, 吕群波, 等. 一种基于频域的序列图像超分辨率增强方法[J]. 光学学报, 2017, 37(7): 83-88.
[10] Zhu F Z, Liu Y, Huang X, et al. Remote sensing image super-resolution based on improved sparse representation[J]. Optics and Precision Engineering, 2019, 27(3): 718-725. 朱福珍, 刘越, 黄鑫, 等. 改进的稀疏表示遥感图像超分辨重建[J]. 光学精密工程, 2019, 27(3): 718-725.
[11] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[C]//LNCS 8689: Proceedings of the 13th European Conference on Computer Visio, Zurich, Sep 6-12, 2014. Berlin, Heidelberg: Springer, 2014: 818-833.
[12] Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-25, 2009. Washington: IEEE Computer Society, 2009: 248-255.
[13] Dong C, Loy C C, He K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 38(2):295-307.
[14] Graves A, Mohamed A, Hinton G E. Speech recognition with deep recurrent neural networks[C]//Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, May 26-31, 2013. Pis-cataway: IEEE, 2013: 6645-6649.
[15] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 770-778.
[16] Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 1637-1645.
[17] Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 27-30, 2016. Washington: IEEE Computer Society, 2016: 1646-1654.
[18] Mao X J, Shen C H, Yang Y B. Image restoration using convolutional auto-encoders with symmetric skip connec-tions[J]. arXiv:1606.08921, 2016.
[19] Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial net-works[J]. arXiv:1511.06434, 2015.
[20] Ledig C, Theis L, Huszar F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, Jul 21-26, 2017.Washington: IEEE Computer Society, 2017: 105-114.
[21] Wang X T, Yu K, Wu S X, et al. ESRGAN: enhanced super-resolution generative adversarial networks[C]//LNCS 11133: Proceedings of the 2018 Workshops on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Heidelberg: Springer, 2018: 63-79.
[22] Yang J, Li W J, Wang R G, et al. Generative adversarial network for image super-resolution combining perceptual loss[J]. Journal of Image and Graphics, 2019, 24(8): 1270-1282.杨娟, 李文静, 汪荣贵, 等. 融合感知损失的生成式对抗超分辨率算法[J]. 中国图象图形学报, 2019, 24(8): 1270-1282.
[23] Zhang Y L, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]//LNCS 11211: Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Berlin, Hei-delberg: Springer, 2018: 294-301.
[24] Li Y F, Fu R D, Jin W, et al. Multi-channel convolution image super-resolution method[J]. Chinese Journal of Image and Graphics, 2018, 22(12): 1690-1700.李云飞, 符冉迪, 金炜, 等. 多通道卷积的图像超分辨率方法[J]. 中国图象图形学报, 2018, 22(12): 1690-1700.
[25] Shao B T, Tang X Y, Jin L, et al. Single frame infrared image super-resolution algorithm based on generative adver-sarial nets[J]. Journal of Infrared and Millimeter Waves, 2018, 37(4): 427-432.邵保泰, 汤心溢, 金璐, 等. 基于生成对抗网络的单帧红外图像超分辨算法[J]. 红外与毫米波学报, 2018, 37(4): 427-432.
[26] Xu J, Liu H, Guo Q, et al. Super-resolution reconstruction of CT images using neural network combined with decon-volution[J]. Journal of Computer-Aided Design & Computer Graphics, 2018, 30(11): 2084-2092.徐军, 刘慧, 郭强, 等. 结合反卷积的CT图像超分辨重建网络[J]. 计算机辅助设计与图形学学报, 2018, 30(11): 2084-2092.
[27] Nair V, Hinton G E. Rectified linear units improve restr-icted Boltzmann machines[C]//Proceedings of the 27th Inter-national Conference on Machine Learning, Haifa, Jun 21-24, 2010. Madison: Omni Press, 2010: 807-814.
[28] Tong T, Li G, Liu X J, et al. Image super-resolution using dense skip connections[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, Oct 22-29, 2017. Washington: IEEE Computer Society, 2017: 4809-4817.
[29] Szegedy C, Liu W, Jia Y Q, et al. Going deeper with convolutions[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Jun 7-12, 2015. Washington: IEEE Computer Society, 2015: 1-9.
[30] Bengio Y, Simard P Y, Frasconi P. Learning long-term dependencies with gradient descent is difficult[J]. IEEE Transactions on Neural Networks, 1994, 5(2): 157-166.
[31] Aharon M, Elad M, Bruckstein A. K-SVD: an algorithm for designing overcomplete dictionaries for sparse represen-tation[J]. IEEE Transactions on Signal Processing, 2006, 54(11): 4311-4322.
[32] Shlens J. A tutorial on principal component analysis[J]. arXiv:1404.1100, 2014.
[33] Ahmed N, Natarajan T, Rao K R. Discrete cosine transform[J]. IEEE Transactions on Computers, 1974, 23(1): 90-93.
[34] Turaga D S, Chen Y W, Caviedes J E. No reference PSNR estimation for compressed pictures[C]//Proceedings of the 2002 International Conference on Image Processing, Roc-hester, Sep 22-25, 2002. Piscataway: IEEE, 2002: 61-64.
[35] Wang Z, Bovik A C, Sheikh H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[36] Zhao H, Gallo O, Frosio I, et al. Loss functions for image restoration with neural networks[J]. IEEE Transactions on Computational Imaging, 2017, 3(1): 47-57.
[37] Kingma D P, Ba J. Adam: a method for stochastic optimization[J]. arXiv:1412.6980, 2014.
[38] Martin D R, Fowlkes C C, Tal D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th International Conference on Computer Vision, Vancouver, Jul 7-14, 2001. Washington: IEEE Computer Society, 2001: 416-425.
[39] Marco B, Aline R, Christine G, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 23rd British Machine Vision Conference, Surrey, Sep 3-7, 2012. Durham: BMVA Press, 2012: 1-10.
[40] Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations[C]//LNCS 6920: Proceedings of the 7th International Conference on Curves and Surfaces, Avignon, Jun 24-30, 2010. Berlin, Heidelberg: Springer, 2010: 711-730.