Improved Two-Branch Capsule Network for Hyperspectral Image Classification

doi:10.3778/j.issn.1673-9418.2102073

Abstract

Abstract:

The method based on the dual-channel capsule network extracts spectral information and spatial informa-tion separately in two channels, which not only retains the feature extraction method of the dual-channel convolu-tional neural network, but also improves the classification accuracy. However, when researchers train the capsule network, the dynamic routing process generates a large number of training parameters because the hyperspectral image (HSI) usually consists of hundreds of channels. To address this limitation, 1D and 2D constraint windows are proposed to reduce the number of capsules from two extraction channels. It uses the capsule vector group as the calculation unit to perform convolution operations and reduce the amount of parameters and computational com-plexity of the capsule network. Based on this parameter reduction optimization method, a new dual-branch capsule neural network (DuB-ConvCapsNet-MRF) is proposed and applied to the task of hyperspectral image classifica-tion. In addition, in order to further improve the classification accuracy, Markov random field (MRF) is introduced to smooth the spatial region and the final output is got. The results of performing ablation experiments on two repre-sentative hyperspectral image datasets and comparing the proposed method with six existing classification methods show that DuB-ConvCapsNet-MRF is superior to other methods in classification performance, and effectively re-duces the cost of training capsule network.

Key words: remote sensing, hyperspectral image classification, capsule neural network, constraint window, Markov random field (MRF)

摘要：

基于双分支的胶囊网络分类方法在两个通道分别提取光谱信息和空间信息,既保留了双分支卷积神经网络的特征提取方式,又提高了分类精度。但由于高光谱图像（HSI）通常由几百个通道组成,在训练胶囊网络时,动态路由过程产生了大量的训练参数。为此提出1D和2D约束窗口分别减少来自两个提取通道的胶囊数量。它以胶囊向量组为计算单位进行卷积运算,来减少胶囊网络的参数量和计算复杂度。基于该降参优化方法提出一个新的双分支胶囊神经网络（DuB-ConvCapsNet-MRF）,并将其应用在高光谱图像分类任务中。此外,为进一步提高分类性能,引入马尔可夫随机场（MRF）对空间区域进行平滑后处理,获得最终输出。对两个代表性高光谱图像数据集进行消融实验并与现有的6个分类方法进行比较,结果表明,DuB-ConvCapsNet-MRF在分类精度上都优于其他方法,并且有效降低了胶囊网络的训练代价。

关键词: 遥感, 高光谱图像分类, 胶囊神经网络, 约束窗口, 马尔可夫随机场（MRF）

CLC Number:

TP751.1

ZHANG Haitao, CHAI Simin. Improved Two-Branch Capsule Network for Hyperspectral Image Classification[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(10): 2405-2414.

张海涛, 柴思敏. 改进双分支胶囊网络的高光谱图像分类[J]. 计算机科学与探索, 2022, 16(10): 2405-2414.

Figures/Tables 16

Fig.1 Network architecture of DuB-ConvCapsNet-MRF

Fig.2 Execution process of dynamic routing

Fig.3 Proposed parameter reduction method, 2D-ConvCaps

Fig.4 Proposed parameter reduction method, 1D-ConvCaps

Fig.5 Indian Pines datasets

Fig.6 University of Pavia datasets

Table 1 Hyper-parameters used for training proposed DuB-ConvCapsNet-MRF

Hyper-parameter	Optimized value
Optimizer	Adam
Number of epochs	100
Dropout	0.2
Learning rate	0.01
Learning rate decay	0.9
PCA	5
$λ$ (in Eq. (7))	0.5
$m +$ (in Eq. (7))	0.9
$m -$ (in Eq. (7))	0.1

Table 1 Hyper-parameters used for training proposed DuB-ConvCapsNet-MRF

Hyper-parameter	Optimized value
Optimizer	Adam
Number of epochs	100
Dropout	0.2
Learning rate	0.01
Learning rate decay	0.9
PCA	5
$λ$ (in Eq. (7))	0.5
$m +$ (in Eq. (7))	0.9
$m -$ (in Eq. (7))	0.1

Fig.7 Impact of the number of channels reserved by PCA on classification performance

Table 2 Classification results of different

Class	SVM	2D-CNN	3D- CNN	DCD-CNN	CNN-MRF	Caps-MRF	Proposed
1	77.88	95.67	98.16	98.45	90.08	89.92	97.72
2	0	94.31	97.86	98.77	96.91	98.49	99.31
3	25.16	91.76	99.85	96.95	97.86	97.62	96.26
4	68.17	81.38	98.13	98.13	96.81	98.84	99.46
5	46.54	98.46	98.64	100.00	97.54	99.91	100.00
6	80.77	99.42	100.00	98.03	99.35	98.87	98.13
7	35.97	96.47	99.46	92.24	92.00	96.56	97.51
8	97.07	100.00	99.51	100.00	99.46	100.00	98.16
9	45.00	62.46	60.00	60.00	68.48	71.79	73.26
10	71.70	95.78	97.89	99.67	98.60	99.78	100.00
11	86.92	97.32	98.67	98.92	97.91	99.17	99.84
12	61.73	93.78	100.00	99.41	98.46	98.82	97.94
13	91.85	97.89	98.87	100.00	100.00	100.00	100.00
14	96.40	99.67	99.46	97.21	95.84	98.26	98.82
15	45.53	87.61	99.21	99.16	100.00	99.16	98.63
16	82.81	88.63	96.16	89.71	85.61	99.25	92.17
OA	77.63	95.13	98.45	98.37	97.15	98.53	98.74
AA	63.34	92.54	96.37	95.36	94.68	96.65	96.70
Kappa	84.21	94.91	98.71	98.23	98.31	98.47	98.93
训练时间/s	12	2 426	33 413	3 120	5 124	21 670	13 268

Table 3 Classification accuracy of models in different situations on IP dataset

MRF	2D-ConvCaps/ 1D-ConvCaps	PCA	OA/%	Kappa/%
√			92.13	90.76
√	√		93.79	92.82
	√	√	97.21	96.48
√	√	√	98.74	98.93

Fig.8 Classification accuracy of different ratios of training samples in IP dataset

Fig.9 Classification maps of different methods on IP dataset

Table 4 Classification accuracy of models in different situations on UP dataset

MRF	2D-ConvCaps/ 1D-ConvCaps	PCA	OA/%	Kappa/%
√			93.94	91.90
√	√		95.76	93.58
	√	√	98.13	97.16
√	√	√	99.61	99.34

Fig.10 Classification accuracy of different ratios of training samples in UP dataset

Table 5 Classification results of different methods on UP dataset

class	SVM	2D-CNN	3D-CNN	DCD-CNN	CNN-MRF	Caps-MRF	Proposed
1	75.45	87.61	99.65	98.75	97.41	99.97	99.46
2	92.41	88.57	99.17	99.16	97.25	98.46	99.43
3	96.23	89.42	94.76	100.00	90.64	98.72	100.00
4	95.76	90.57	99.29	98.13	96.71	99.67	99.71
5	98.01	90.64	100.00	99.87	99.18	99.16	99.13
6	93.98	70.67	98.74	100.00	98.34	99.43	98.87
7	82.71	88.96	96.96	100.00	98.74	99.56	100.00
8	64.64	89.79	99.26	99.34	90.74	99.81	99.67
9	97.81	91.13	99.16	99.16	99.28	98.46	99.38
OA	75.69	87.46	98.91	99.57	96.16	99.24	99.61
AA	88.56	87.48	98.55	99.38	96.48	99.25	99.52
Kappa	70.04	85.17	99.22	99.26	95.33	97.98	99.34
训练时间/s	46	7 261	100 496	9 346	14 962	63 570	39 715

Table 6 Amount of parameters brought by different number of vector groups for input and feature extraction

$a 1$	$a 2$
$a 1$	4	8	16	32	64	未降参
64	354 816	709 632	1 419 264	2 838 528	5 677 056	4 956 160
32	281 088	562 176	1 124 352	2 248 704	4 497 408	2 478 080
16	244 224	488 448	976 896	1 953 792	3 907 584	1 239 040

Table 6 Amount of parameters brought by different number of vector groups for input and feature extraction

$a 1$	$a 2$
$a 1$	4	8	16	32	64	未降参
64	354 816	709 632	1 419 264	2 838 528	5 677 056	4 956 160
32	281 088	562 176	1 124 352	2 248 704	4 497 408	2 478 080
16	244 224	488 448	976 896	1 953 792	3 907 584	1 239 040

References 27

[1]	CAO X H, LI R J, GE Y M, et al. Densely connected deep random forest for hyperspectral imagery classification[J]. Inter-national Journal of Remote Sensing, 2019, 40(9): 3606-3622.
[2]	LANTHIER Y, BANNARI A, HABOUDANE D, et al. Hy-perspectral data segmentation and classification in preci-sion agriculture: a multi-scale analysis[C]// Proceedings of the 2008 IEEE International Geoscience & Remote Sensing Symposium, Boston, Jul 8-11, 2008. Piscataway: IEEE, 2008: 585-588.
[3]	GHAMISI P, YOKOYA N, LI J, et al. Advances in hyper-spectral image and signal processing: a comprehensive over-view of the state of the art[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4): 37-78. DOI URL
[4]	WANG Q, LIU S T, CHANUSSOT J, et al. Scene classifi-cation with recurrent attention of VHR remote sensing ima-ges[J]. IEEE Transactions on Geoscience and Remote Sen-sing, 2018, 57(2): 1155-1167.
[5]	LANDGREBE D. Hyperspectral image data analysis[J]. IEEE Signal Processing Magazine, 2002, 19(1): 17-28.
[6]	FAUVEL M, TARABALKA Y, BENEDIKTSSON J A, et al. Advances in spectral-spatial classification of hyperspectral images[J]. Proceedings of the IEEE, 2012, 101(3): 652-675. DOI URL
[7]	HAM J, CHEN Y, CRAWFORD M M, et al. Investigation of the random forest framework for classification of hyper-spectral data[J]. IEE20E Transactions on Geoscience and Remote Sensing, 2005, 43(3): 492-501.
[8]	CHEN Y, NASRABADI N M, TRAN T D. Hyperspectral image classification using dictionary-based sparse represen-tation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(10): 3973-3985. DOI URL
[9]	HAUT J M, PAOLETTI M, PLAZA J, et al. Cloud implemen-tation of the K-means algorithm for hyperspectral image analysis[J]. The Journal of Supercomputing, 2017, 73(1): 514-529. DOI URL
[10]	LIU W B, WANG Z D, LIU X H, et al. A survey of deep neural network architectures and their applications[J]. Neuro-computing, 2017, 234: 11-26.
[11]	CHEN Y S, LIN Z H, ZHAO X, et al. Deep learning-based classification of hyperspectral data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(6): 2094-2107. DOI URL
[12]	CHEN Y S, ZHAO X, JIA X P. Spectral-spatial classifica-tion of hyperspectral data based on deep belief network[J]. IEEE Journal of Selected Topics in Applied Earth Obser-vations and Remote Sensing, 2015, 8(6): 2381-2392.
[13]	万亚玲, 钟锡武, 刘慧, 等. 卷积神经网络在高光谱图像分类中的应用综述[J]. 计算机工程与应用, 2021, 57(4): 1-10. DOI
	WAN Y L, ZHONG X W, LIU H, et al. Survey of applica-tion of convolutional neural network in classification of hyperspectral images[J]. Computer Engineering and Appli-cations, 2021, 57(4): 1-10.
[14]	MAKANTASIS K, KARANTZALOS K, DOULAMIS A D, et al. Deep supervised learning for hyperspectral data classi-fication through convolutional neural networks[C]// Procee-dings of the 2015 IEEE International Geoscience and Re-mote Sensing Symposium, Milan, Jul 26-31, 2015. Piscata-way: IEEE, 2015: 4959-4962.
[15]	HAMIDA A B, BENOIT A, LAMBERT P, et al. 3-D deep learning approach for remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(8): 4420-4434. DOI URL
[16]	佘海龙, 解山娟, 邹静洁. 标准分数降维的3D-CNN高光谱遥感图像分类[J]. 计算机工程与应用, 2021, 57(4): 169-175. DOI
	SHE H L, XIE S J, ZOU J J. 3D-CNN with standard score dimensionality reduction for hyperspectral remote sensing images classification[J]. Computer Engineering and Appli-cations, 2021, 57(4): 169-175.
[17]	XU X D, LI W, RAN Q, et al. Multisource remote sensing data classification based on convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 56(2): 937-949. DOI URL
[18]	刘启超, 肖亮, 刘芳, 等. SSCDenseNet: 一种空-谱卷积稠密网络的高光谱图像分类算法[J]. 电子学报, 2020, 48(4): 751-762. DOI
	LIU Q C, XIAO L, LIU F, et al. SSCDenseNet: a spectral-spatial convolutional dense network for hyperspectral image classification[J]. Acta Electronica Sinica, 2020, 48(4): 751-762.
[19]	SABOUR S, FROSST N, HINTON G E. Dynamic routing between capsules[J]. arXiv:1710.09829, 2017.
[20]	JIANG X F, LIU W B, ZHANG Y, et al. Spectral-spatial hy-perspectral image classification using dual-channel capsule networks[J]. IEEE Geoscience and Remote Sensing Let-ters, 2021, 18(6): 1094-1098.
[21]	JIANG X F, ZHANG Y, LIU W B, et al. Hyperspectral ima-ge classification with CapsNet and Markov random fields[J]. IEEE Access, 2020, 8: 191956-191968. DOI URL
[22]	ZHANG H T, MENG L G, WEI X, et al. 1D-convolutional capsule network for hyperspectral image classification[J]. arXiv:1903.09834, 2019.
[23]	TARABALKA Y, FAUVEL M, CHANUSSOT J, et al. SVM- and MRF-based method for accurate classification of hy-perspectral images[J]. IEEE Geoscience and Remote Sen-sing Letters, 2010, 7(4): 736-740.
[24]	YUE J, ZHAO W Z, MAO S J, et al. Spectral-spatial clas-sification of hyperspectral images using deep convolutio-nal neural networks[J]. Remote Sensing Letters, 2015, 6(6): 468-477. DOI URL
[25]	LI Y, ZHANG H K, SHEN Q. Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network[J]. Remote Sensing, 2017, 9(1): 67. DOI URL
[26]	胡丽, 单锐, 王芳, 等. 基于双通道空洞卷积神经网络的高光谱图像分类[J]. 激光与光电子学进展, 2020, 57(12): 348-354.
	HU L, SHAN R, WANG F, et al. Hyperspectral image clas-sification based on dual-channel dilated convolution neural network[J]. Laser & Optoelectronics Progress, 2020, 57(12): 348-354.
[27]	CAO X Y, ZHOU F, XU L, et al. Hyperspectral image classi-fication with Markov random fields and a convolutional neural network[J]. IEEE Transactions on Image Processing, 2018, 27(5): 2354-2367. DOI URL