Low-Light Enhancement Method for Light Field Images by Fusing Multi-scale Features

doi:10.3778/j.issn.1673-9418.2202064

Abstract

Abstract: Light field images (LFI) record rich 3D structural and textural details of target scenes, which have great advantages in a wide range of computer vision tasks. However, LFI captured under low-light conditions always suffer from low brightness and strong noise, which may seriously degrade the quality of LFI. In this paper, a low-light LFI enhancement method by fusing multi-scale light field (LF) structural features is proposed , which adopts digital single-lens reflex camera (DSLR) images to supervise the generated enlightened LFI. To explore and exploit light field structural features, angular and spatial Transformers are introduced to extract LF structural features from LFI at different scales, i.e., the complementary information between sub-views and local and long-range dependencies within each sub-view. A recurrent fusion module is proposed to preserve the long-term memory of features at diffe-rent scales by using a long-short-term memory network, while adaptively aggregating LF structural features in the entire feature space through local and global fusion layers. A 4D residual reconstruction module is designed to reconstruct target LFI sub-views from the aggregated features. In additional, a dataset of low-light LFI and normal-light DSLR image pairs is constructed to train the proposed network. Extensive experiment demonstrates that the proposed network can effectively improve the quality of low-light LFI, and it obviously outperforms other state-of-the-art methods.

Key words: light field images (LFI), low-light image enhancement, Transformer, 4D convolution

摘要： 光场图像（LFI）记录了目标场景丰富的三维结构和纹理等信息，在多种计算机视觉任务中拥有巨大优势。但是，低光照条件下采集的光场图像存在亮度低、噪声大等问题，降低了图像质量。提出了一种多尺度特征融合的低照度光场图像增强算法，引入数码单反相机（DSLR）图像来监督网络的训练以提升低照度光场图像的质量。为了充分挖掘和利用光场信息，通过角度和空间Transformer在不同尺度上对光场图像进行特征提取，捕获每个子孔径图像的互补信息以及局部和远程依赖关系。提出一个循环融合模块，利用长短时记忆网络保留不同尺度特征的长时记忆，同时通过局部和全局融合层自适应地聚合整个特征空间中的有用信息。设计了一个4D残差模块从聚合的特征重建目标光场子视图。此外，还构建了一个低照度LFI 和正常光照DSLR图像配对的数据集来训练所提出的网络。实验结果表明，所提网络能够有效地提升低照度光场图像的质量，相比其他算法拥有明显的优势。

关键词: 光场图像（LFI）, 低照度图像增强, Transformer, 4D卷积

LI Mingyue, YAN Tao, JING Huahua, LIU Yuan. Low-Light Enhancement Method for Light Field Images by Fusing Multi-scale Features[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(8): 1904-1916.

李明悦, 晏涛, 井花花, 刘渊. 多尺度特征融合的低照度光场图像增强算法[J]. 计算机科学与探索, 2023, 17(8): 1904-1916.

References

[1] KALANTARI N K, WANG T C, RAMAMOORTHI R. Lear-ning-based view synthesis for light field cameras[J]. ACM Transactions on Graphics, 2016, 35(6): 1-10.
[2] LI T, LUN D P K, CHAN Y H. Robust reflection removal based on light field imaging[J]. IEEE Transactions on Image Processing, 2018, 28(4): 1798-1812.
[3] ZHANG M, JI W, PIAO Y, et al. LFNet: light field fusion network for salient object detection[J]. IEEE Transactions on Image Processing, 2020, 29: 6276-6287.
[4] ZHU H, ZHANG Q, WANG Q. 4D light field superpixel and segmentation[J]. IEEE Transactions on Image Processing, 2020, 29: 85-99.
[5] SI L, WANG Q. Dense depth-map estimation and geometry inference from light fields via global optimization[C]//LNCS 10113: Proceedings of the 13th Asian Conference on Computer Vision, Taipei, China, Nov 21-23, 2016. Cham: Springer, 2016: 83-98.
[6] CHENG H D, SHI X J. A simple and effective histogram equalization approach to image enhancement[J]. Digital Signal Processing, 2004, 14(2): 158-170.
[7] SHEET D, GARUD H, SUVEER A, et al. Brightness pre-serving dynamic fuzzy histogram equalization[J]. IEEE Tran-sactions on Consumer Electronics, 2010, 56(4): 2475-2480.
[8] YUN S H, KIM J H, KIM S. Contrast enhancement using a weighted histogram equalization[C]//Proceedings of the 2011 IEEE International Conference on Consumer Electronics, Las Vegas, Jan 9-12, 2011. Piscataway: IEEE, 2011: 203-204.
[9] HUANG S C, CHENG F C, CHIU Y S. Efficient contrast enhancement using adaptive gamma correction with weigh-ting distribution[J]. IEEE Transactions on Image Processing, 2012, 22(3): 1032-1041.
[10] WANG W, SUN N, NG M K. A variational gamma correction model for image contrast enhancement[J]. Inverse Problems & Imaging, 2019, 13(3): 461.
[11] LAND E H, MCCANN J J. Lightness and retinex theory[J].Journal of the Optical Society of America, 1971, 61(1): 1-11.
[12] JOBSON D J, RAHMAN Z, WOODELL G A. Properties and performance of a center/surround retinex[J]. IEEE Tran-sactions on Image Processing, 1997, 6(3): 451-462.
[13] RAHMAN Z, JOBSON D J, WOODELL G W. A multiscale retinex for color rendition and dynamic range compression[C]//Proceedings of SPIE-The International Society for Optical Engineering, Denver, Aug 4, 1996. San Francisco: SPIE, 1996: 183-191.
[14] GUO X, LI Y, LING H. LIME: low-light image enhance-ment via illumination map estimation[J]. IEEE Transac-tions on Image Processing, 2016, 26(2): 982-993.
[15] LI M, LIU J, YANG W, et al. Structure-revealing low-light image enhancement via robust retinex model[J]. IEEE Tran-sactions on Image Processing, 2018, 27(6): 2828-2841.
[16] LV F, LU F, WU J, et al. MBLLEN: low-light image/video enhancement using CNNs[C]//Proceedings of the British Machine Vision Conference 2018, Newcastle, Sep 3-6, 2018. Durham: BMVA Press, 2018: 220.
[17] 江泽涛, 覃露露. 一种基于U-Net生成对抗网络的低照度图像增强方法[J]. 电子学报, 2020, 48(2): 258-264.
JIANG Z T, QIN L L. Low-light image enhancement method based on U-Net generative adversarial network[J]. Acta Elec-tronica Sinica, 2020, 48(2): 258-264.
[18] XU K, YANG X, YIN B, et al. Learning to restore low-light images via decomposition-and-enhancement[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 2278-2287.
[19] WEI C, WANG W, YANG W, et al. Deep retinex decom-position for low-light enhancement[J]. arXiv:1808.04560, 2018.
[20] 马红强, 马时平, 许悦雷, 等. 基于深度卷积神经网络的低照度图像增强[J]. 光学学报, 2019, 39(2): 91-100.
MA H Q, MA S P, XU Y L, et al. Low-light image enhance-ment based on deep convolutional neural network[J]. Acta Optica Sinica, 2019, 39(2): 91-100.
[21] 江泽涛, 覃露露, 秦嘉奇, 等. 一种基于 MDARNet 的低照度图像增强方法[J]. 软件学报, 2021, 32(12): 3977-3991.
JIANG Z T, QIN L L, QIN J Q, et al. Low-light image enhancement method based on MDARNet[J]. Journal of Software, 2021, 32(12): 3977-3991.
[22] 潘晓英, 魏苗, 王昊, 等. 多尺度融合残差编解码器的低照度图像增强方法[J]. 计算机辅助设计与图形学学报, 2022, 34(1): 104-112.
PAN X Y, WEI M, WANG H, et al. A multi-scale fusion residual encoder-decoder approach for low illumination image enhancement[J]. Journal of Computer-Aided Design & Com-puter Graphics, 2022, 34(1): 104-112.
[23] LV F, LI Y, LU F. Attention guided low-light image enhan-cement with a large scale low-light simulation dataset[J]. arXiv:1908.00682, 2019.
[24] 王克琪, 钱宇华, 梁吉业, 等. 局部-全局关系耦合的低照度图像增强[J]. 中国科学(信息科学), 2022, 52(3): 443-460.
WANG K Q, QIAN Y H, LIANG J Y, et al. Local-global coupling relationship based low-light image enhancement[J]. Science in China (Information Sciences), 2022, 52(3): 443-460.
[25] ZHANG Z, JIANG Y, JIANG J, et al. STAR: a structure-aware lightweight transformer for real-time image enhance-ment[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, Oct 10-17, 2021. Piscataway: IEEE, 2021: 4106-4115.
[26] GUO C, LI C, GUO J, et al. Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 1780-1789.
[27] JIANG Y, GONG X, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing, 2021, 30: 2340-2349.
[28] ZHENG S, GUPTA G. Semantic-guided zero-shot learning for low-light image/video enhancement[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 4-8, 2022. Piscataway: IEEE, 2022: 581-590.
[29] LAMBA M, RACHAVARAPU K K, MITRA K. Harnes-sing multi-view perspective of light fields for low-light ima-ging[J]. IEEE Transactions on Image Processing, 2020, 30: 1501-1513.
[30] LAMBA M, MITRA K. Fast and efficient restoration of extremely dark light fields[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, Jan 3-8, 2022. Piscataway: IEEE, 2022: 1361-1370.
[31] ZHANG S, LAM E Y. An effective decomposition-enhance-ment method to restore light field images captured in the dark[J]. Signal Processing, 2021, 189: 108279.
[32] LIANG Z, WANG Y, WANG L, et al. Light field image super-resolution with transformers[J]. IEEE Signal Proces-sing Letters, 2022, 29: 563-567.
[33] ZHANG H, PATEL V M. Density-aware single image de-raining using a multi-stream dense network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, Jun 18-23, 2018. Piscataway: IEEE, 2018: 695-704.
[34] JIANG K, WANG Z, YI P, et al. Multi-scale progressive fusion network for single image deraining[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 13-19, 2020. Piscataway: IEEE, 2020: 8346-8355.
[35] YAN Q, ZHANG L, LIU Y, et al. Deep HDR imaging via a non-local network[J]. IEEE Transactions on Image Proces-sing, 2020, 29: 4308-4322.
[36] SHI X, CHEN Z, WANG H, et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting[C]//Advances in Neural Information Proces-sing Systems 28, Montreal, Dec 7-12, 2015: 802-810.
[37] ZHANG S, GUO S, HUANG W, et al. V4D: 4D convolutional neural networks for video-level representation learning[J]. arXiv:2002.07442, 2020.
[38] SHEN X, DARMON F, EFROS A A, et al. RANSAC-Flow: generic two-stage image alignment[C]//LNCS 12349: Procee-dings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 618-637.
[39] MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2012, 20(3): 209-212.
[40] MITTAL A, MOORTHY A K, BOVIK A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708.