关键词: 数据虚拟化, 模型驱动, 数据湖, 分布式

Abstract: In view of the characteristics of wide distribution, multiple types and strong uncertainty of business service objects of the Federation of Industry and Commerce, a distributed data lake construction method driven by virtualization model is proposed. The architecture of distributed data lake is given. The data virtualization model adapted to scattered and fragmented data collection scenarios and model-driven inter-database collaboration process are defined. Based on this model and the virtualized global data indexing network, the routing and coordination among edge database nodes, secondary regional database nodes and central database nodes are realized, and the distributed spoke-type data lake which can be de-ETL and de-centralization is formed. The proposed method can alleviate the problems of centralized data lake, such as poor timeliness of data updating, large storage demand, high bandwidth usage, low economy and so on. Comparisons show that the proposed method can meet the needs of large data analysis and data real-time processing of the Federation of Industry and Commerce, reducing data handling costs and improving economy.

Key words: data virtualization, model driven, data lake, distributed