Journal of Frontiers of Computer Science and Technology ›› 2018, Vol. 12 ›› Issue (3): 360-369.DOI: 10.3778/j.issn.1673-9418.1612024

Previous Articles     Next Articles

Data Source Selection for Web Big Data System

LIU Zhengtao1,2+, WANG Jiandong2   

  1. 1. College of Computer Science and Engineering, Sanjiang University, Nanjing 210012, China
    2. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China
  • Online:2018-03-01 Published:2018-03-08

Web大数据系统数据源选择

刘正涛1,2+,王建东2   

  1. 1. 三江学院 计算机科学与工程学院,南京 210012
    2. 南京航空航天大学 计算机科学与技术学院,南京 210016

Abstract:  How to select the appropriate data source from the large number of Web data sources, so as to reduce the number of accessing data sources, is one of the key issues in the integration of Web big data system. This paper proposes a two-stage data source selection method. The first stage is to select the data source with the high similarity to the middle schema and select the data source with the high reliability by computing the quality of dependent data source. In the second stage, a time-cost minimization query algorithm is designed for source permutation. To calculate the repetition rate of the data source, the maximum entropy theory is applied in the algorithm. Finally, the algorithm is evaluated on the experimental platform. The experiments show that the proposed algorithm has high efficiency and scalability compared with other algorithms.

Key words: Web big data, data source selection, quality of data source, dependence of data source

摘要: 如何从数量众多的Web数据源集合中选择数量合适的数据源,使得在满足特定查询需求的前提下尽可能地减少访问数据源的数量,是Web大数据系统集成中的关键问题之一。提出了一个两阶段数据源选择方案:第一阶段通过各个数据源模式与中间模式的相似度选择与查询相关度高的数据源,通过计算依赖数据源的质量来选取质量较好的数据源;第二阶段基于最大熵理论计算数据源之间的重复率,设计实现了一个查询最小代价模型动态选择数据源算法。最后在实验平台上对算法进行了评估,实验表明该算法具有较高的效率与扩展性。

关键词: Web大数据, 数据源选择, 数据源质量, 数据源依赖