Distributed Storage Method for Equipment Data Based on Pre-partitioning Strategy

doi:10.3778/j.issn.1673-9418.2002019

Abstract

Abstract:

With the development of sensor technology and computer technology, equipment can produce a large amount of data during the development and production process. These data are massive, multi-source, and heterogeneous. Enterprises need to consider how to effectively manage large amount of equipment data and use processed data to enhance manufacturing capabilities. This paper studies the data of typical equipment, such as satellite, airplane, etc., and proposes a distributed data storage method based on a pre-partitioning strategy. Through studying the pre-partitioning mechanism of HBase and the characteristics of equipment data, this paper studies impact factors for rapid storage of equipment data, and proposes a fast storage algorithm, which can store large amount of data into HBase in a balanced and fast manner. Finally, this paper evaluates the data storage performance, load balance, and applicability of various types of equipment. The experimental results show that this method can be applied to many types of equipment and has good performance in data storage efficiency.

Key words: equipment, data storage, distributed column database, pre-partitioning strategy

摘要：

随着传感器技术和计算机技术的发展，装备在研制生产过程中会产生大量的数据，这些数据是海量的、多源的和异构的，企业需要考虑如何将数据进行快速处理和存储管理，进而利用加工后的数据提升装备生产制造能力。对卫星、飞机等典型装备数据进行了研究，提出了一种基于预分区策略的分布式数据存储方法。该方法研究HBase的预分区机制和装备数据模型特点，研究装备数据快速存储的影响因子，并给出了数据快速存储算法，使海量装备数据可以负载均衡地、快速地存储在HBase数据库里。最后，对模型的数据存储性能、负载均衡性、各类装备的适用性进行了评估试验。试验结果表明，该方法可以覆盖多种类型的装备数据，并在数据存储效率上有良好的表现。

关键词: 装备, 数据存储, 分布式列数据库, 预分区策略

GAO Jian, WEI Jun, XU Lijie, WANG Baolong, YANG Fuxue, HUANG Xiaofei. Distributed Storage Method for Equipment Data Based on Pre-partitioning Strategy[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(1): 96-108.

高健, 魏峻, 许利杰, 汪保龙, 杨富学, 黄骁飞. 基于预分区策略的装备数据分布式存储方法[J]. 计算机科学与探索, 2021, 15(1): 96-108.

References

[1] PEVEC D, VDOVIC H,?GACE I, et al.?Distributed data plat-form for automotive industry: a robust solution for tackling big challenges of big data in transportation science[C]//Proceedings of the 2019 15th International Conference on Telecommunications, Graz, Jul 3-5, 2019. Piscataway: IEEE, 2019: 1-8.
[2] WANG Z H, LIANG Z Y, LI J Z. Survey on industrial big data analysis: models and algorithms[J]. Big Data, 2018, 4(5): 62-79.
[3] Industrial Big Data Task Group of Alliance of Industrial In-ternet. Technology and application practice of industrial big data[M]. Beijing: Publishing House of Electronics Industry, 2014.
工业大数据特设组. 工业大数据技术与应用实践[M]. 北京: 电子工业出版社, 2014.
[4] BRENNER M J, LIND R C, VORACEK D F, et al. Over-view of recent flight flutter testing research at NASA dryden[C]//Proceedings of the 38th Structures, Structural Dynamics, and Materials Conference, Kissimmee, Apr 7-10, 1997. Ame-rican Institute of Aeronautics and Astronautics, 1997: 24-35.
[5] SU L, ZHANG B W, CHEN M, et al. Development and app-lication of big data technology in aerospace field[J]. Jour-nal of Telemetry, Tracking and Command, 2015, 36(2): 1-9.
苏丽, 张博为, 谌明, 等. 大数据技术在航天领域发展与应用[J]. 遥测遥控, 2015, 36(2): 1-9.
[6] KAZI S I. Role of avionics full duplex ethernet protocol in an?aircraft?data communication network[C]//Proceedings of the 2013 International Conference on Green Computing, Com-munication and Conservation of Energy, Chennai, Jun 2, 2014. Piscataway: IEEE, 2014: 706-712.
[7] JOSE J, VARGHESE S. Design of 1553?protocol?controller for reliable?data?transfer in?aircrafts[C]//Proceedings of the 12th International Conference on Intelligent Systems Design and Applications, Kochi, Nov 27-29, 2012. Piscataway: IEEE, 2012: 686-691.
[8] METCALF J, PETERSON L, CARRASQUILLO R, et al. National aeronautics and space administration (NASA) env-ironmental control and life support (ECLS) integrated road-map development[C]//Proceedings of the 42nd Interna-tional Conference on Environmental Systems, San Diego, Jul 15-19, 2012. American Institute of Aeronautics and Ast-ronautics, 2012: 1-7.
[9] CHANG F, DEAN J, GHEMAWAT S. Bigtable: a distributed storage system for structured data[J]. ACM Transactions on Computer Systems, 2008, 26(2): 4.
[10] PIKE R, DORWARD S, GRIESEMER R, et al. Interpreting the data: parallel analysis with Sawzall[J]. Scientific Pro-gramming, 2005, 13(4): 277-298.
[11] ZHAO Y, LI C, XIAO K, et al. Research on distributed storage method of geological and mineral big data based on Hadoop[J]. Geological Bulletin of China, 2019, 38(2): 462-470.
[12] FENG C, LI B. Research of temporal information index strategy based on HBase[J]. Procedia Computer Science, 2017, 107: 367-372.
[13] BHUPATHIRAJU V, RAVURI R P. The dawn of big data- Hbase[C]//Proceedings of the 2014 Conference on IT in Busi-ness, Industry and Government, Indore, Mar 8-9, 2014. Pis-cataway: IEEE, 2015: 1-6.
[14] DING F, CHEN C S, ZHANG T, et al. Research and imple-mentation of coprocessor based HBase region-level secon-dary index[J]. Journal of Computer Applications, 2014(Z1): 181-185.
丁飞, 陈长松, 张涛, 等. 基于协处理器的HBase区域级第二索引研究与实现[J]. 计算机应用, 2014(Z1): 181-185.
[15] SALOUSTROS G, MAGOUTIS K. Rethinking HBase: de-sign and implementation of an elastic key-value store over log-structured local volumes[C]//Proceedings of the 14th International Symposium on Parallel and Distributed Com-puting, Limassol, Jun 29-Jul 2, 2015. Washington: IEEE Com-puter Society, 2015: 225-234.
[16] GAREFALAKIS P, PAPADOPOULOS P, MAGOUTIS K. ACaZoo: a distributed key-value store based on replicated LSM-trees[C]//Proceedings of the 2014 IEEE International Symposium on Reliable Distributed Systems, Nara, Oct 6-9, 2014. Washington: IEEE Computer Society, 2014: 211-220.
[17] HE S W, CHU L X, LI X Y. Spatial query processing for location based application on Hbase[C]//Proceedings of the 2017 IEEE 2nd International Conference on Big Data Analysis, Beijing, Mar 10-12, 2017. Piscataway: IEEE, 2017: 110-114.
[18] CAO C, WANG W Y, ZHANG Y, et al. Leveraging column family to improve multidimensional query performance in HBase[C]//Proceedings of the 2017 IEEE 10th International Conference on Cloud Computing, Honolulu, Jun 25-30, 2017. Washington: IEEE Computer Society, 2017: 106-113.
[19] HUAI Y, MA S Y, LEE R B, et al. Understanding insights into the basic structure and essential issues of table place-ment methods in clusters[J]. Proceedings of the VLDB End-owment, 2013, 6(14): 1750-1761.
[20] NISHIMURA S, DAS S, AGRAWAL D, et al. MD-HBase: design and implementation of an elastic data infrastructure for cloud-scale location services[J]. Distributed and Parallel Databases, 2013, 31(2): 289-319.
[21] RICHTER S, QUIANE?-RUIZ J A, SCHUH S, et al. Tow-ards zero-overhead adaptive indexing in Hadoop[J]. arXiv:1212.3480, 2012.
[22] DITTRICH J, QUIANé-RUIZ J A, RICHTER S, et al. Only aggressive elephants are fast elephants[J]. Proceedings of the VLDB Endowment, 2012, 5(11): 1591-1602.
[23] VASHISHTHA H, STROULIA E. Enhancing query support in HBase via an extended coprocessor framework[C]//LNCS 6994: Proceedings of the 4th European Conference on To-wards a Service-Based Internet, Poznan, Oct 26-28, 2011. Berlin, Heidelberg: Springer, 2011: 75-87.
[24] YANG F Z, MILOSEVIC D, CAO J. An evolutionary algo-rithm for column family schema optimization in HBase[C]//Proceedings of the 1st IEEE International Conference on Big Data Computing Service and Applications, Redwood City, Mar 30-Apr 2, 2015. Washington: IEEE Computer Society, 2015: 439-445.
[25] GAO H B, WANG D F. LSD2H: a novel storage method of linked sensor data based on HBase[C]//Proceedings of the 2014 10th International Conference on Semantics, Knowl-edge and Grids, Beijing, Aug 27-29, 2014. Washington: IEEE Computer Society, 2014: 116-119.