关键词: 大数据, 数据分析, 数据处理, 数据平台, Hadoop

Abstract: The quick development of open source software for big data makes it possible for small and medium businesses and organizations to build and operate their own data platform. However, there are still many challenges in practice such as how to share not only the data but the computing seamlessly, as well as keep users' source code under control without any harm to platform and data. This paper provides a design of data platform which can share the data and computing easily, at the same time, users' source code is also hosted with version control and under scrutiny to ensure the safety. The design is implemented with open source software on real hardware in production. The basic benchmark shows that it works as expected. The platform is open to public and successfully supports the data sharing and computing of the Shanghai Open Data Apps (SODA) competition.

Key words: big data, big data analysis, big data processing, data platform, Hadoop