计算机科学与探索 ›› 2017, Vol. 11 ›› Issue (10): 1557-1569.DOI: 10.3778/j.issn.1673-9418.1609010

• 数据库技术 • 上一篇    下一篇

高维相关性缺失数据的分块填补算法研究

杨  杰1+,杨  虎1,王鲁滨1,金  鑫1,郭  华2,于亮亮3   

  1. 1. 中央财经大学 信息学院,北京 100081
    2. 国网荆州供电公司 信通分公司,湖北 荆州 434000
    3. 国网辽宁省电力有限公司 信息通信分公司,沈阳 110000
  • 出版日期:2017-10-01 发布日期:2017-10-20

Research on Block Imputation Algorithm for High Dimensional Correlation Missing Data

YANG Jie1+, YANG Hu1, WANG Lubin1, JIN Xin1, GUO Hua2, YU Liangliang3   

  1. 1. School of Information, Central University of Finance and Economics, Beijing 100081, China
    2. Jingzhou Power Supply Company ICT Branch of State Grid Corporation, Jingzhou, Hubei 434000, China
    3. Liaoning Power Supply Company ICT Branch of State Grid Corporation, Shenyang 110000, China
  • Online:2017-10-01 Published:2017-10-20

摘要: 研究了高维相关性缺失数据的填补方法,提出了分块填补算法。该算法核心思想是:在填补数据的过程中会考虑变量之间的相互关系,仅利用与待填补数据有相关性的数据进行填补,从而降低不相关数据对缺失数据填补的影响,提高数据填补的准确度。同时,该算法能够并行处理缺失数据,从而提高数据填补效率,对于高维缺失数据的填补有重要意义。为了对分块情况未知的缺失数据进行分块,提出了基于k-means聚类的分块算法。大量的仿真实验和基于真实数据集的实验表明,对于相关性数据,分块填补算法能够有效地利用相关信息进行填补,从而提高数据填补准确度。

关键词: 高维相关性数据, 缺失数据, 分块填补算法

Abstract: This paper studies the method of filling the high dimensional correlation missing data, and proposes a new imputation algorithm based on data block. The key idea of the algorithm is to consider the correlation between variables when filling missing data, and only use the data correlated with the missing data to fill, thereby reducing imputation effects of the missing data caused by the irrelevant data, and improving the accuracy of data imputation. At the same time, the proposed imputation algorithm can be implemented in a parallel way, so that it performs efficiently to fill the high dimensional missing data. In order to divide the missing data with unknown information about blocks into several blocks, this paper proposes a block algorithm based on k-means clustering. Simulation     research and application show that the proposed imputation algorithm is more effective and accurate to handle the missing for the correlation high dimensional data with considering variables?? block relationship than others with not.

Key words: high dimensional correlation data, missing data, block imputation algorithm