关键词: Web大数据, 数据源选择, 数据源质量, 数据源依赖

Abstract:  How to select the appropriate data source from the large number of Web data sources, so as to reduce the number of accessing data sources, is one of the key issues in the integration of Web big data system. This paper proposes a two-stage data source selection method. The first stage is to select the data source with the high similarity to the middle schema and select the data source with the high reliability by computing the quality of dependent data source. In the second stage, a time-cost minimization query algorithm is designed for source permutation. To calculate the repetition rate of the data source, the maximum entropy theory is applied in the algorithm. Finally, the algorithm is evaluated on the experimental platform. The experiments show that the proposed algorithm has high efficiency and scalability compared with other algorithms.

Key words: Web big data, data source selection, quality of data source, dependence of data source