计算机科学与探索 ›› 2021, Vol. 15 ›› Issue (1): 96-108.DOI: 10.3778/j.issn.1673-9418.2002019

• 系统软件与软件工程 • 上一篇    下一篇

基于预分区策略的装备数据分布式存储方法

高健,魏峻,许利杰,汪保龙,杨富学,黄骁飞   

  1. 1. 中国科学院 软件研究所 软件工程技术研发中心,北京 100190
    2. 中国科学院大学,北京 100049
    3. 北京电子工程总体研究所,北京 100039
  • 出版日期:2021-01-01 发布日期:2021-01-07

Distributed Storage Method for Equipment Data Based on Pre-partitioning Strategy

GAO Jian, WEI Jun, XU Lijie, WANG Baolong, YANG Fuxue, HUANG Xiaofei   

  1. 1. Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China
    2. University of Chinese Academy of Sciences, Beijing 100049, China
    3. Beijing General Institute of Electronic Engineering, Beijing 100039, China
  • Online:2021-01-01 Published:2021-01-07

摘要:

随着传感器技术和计算机技术的发展,装备在研制生产过程中会产生大量的数据,这些数据是海量的、多源的和异构的,企业需要考虑如何将数据进行快速处理和存储管理,进而利用加工后的数据提升装备生产制造能力。对卫星、飞机等典型装备数据进行了研究,提出了一种基于预分区策略的分布式数据存储方法。该方法研究HBase的预分区机制和装备数据模型特点,研究装备数据快速存储的影响因子,并给出了数据快速存储算法,使海量装备数据可以负载均衡地、快速地存储在HBase数据库里。最后,对模型的数据存储性能、负载均衡性、各类装备的适用性进行了评估试验。试验结果表明,该方法可以覆盖多种类型的装备数据,并在数据存储效率上有良好的表现。

关键词: 装备, 数据存储, 分布式列数据库, 预分区策略

Abstract:

With the development of sensor technology and computer technology, equipment can produce a large amount of data during the development and production process. These data are massive, multi-source, and heterogeneous. Enterprises need to consider how to effectively manage large amount of equipment data and use processed data to enhance manufacturing capabilities. This paper studies the data of typical equipment, such as satellite, airplane, etc., and proposes a distributed data storage method based on a pre-partitioning strategy. Through studying the pre-partitioning mechanism of HBase and the characteristics of equipment data, this paper studies impact factors for rapid storage of equipment data, and proposes a fast storage algorithm, which can store large amount of data into HBase in a balanced and fast manner. Finally, this paper evaluates the data storage performance, load balance, and applicability of various types of equipment. The experimental results show that this method can be applied to many types of equipment and has good performance in data storage efficiency.

Key words: equipment, data storage, distributed column database, pre-partitioning strategy