关键词: 日志处理, 事件流, 大数据分析, 高性能计算

Abstract: CNGrid is a large high-performance computing environment integrated by many clusters from supercomputing centers, research institutes and universities in China. It provides high quality computing resources to researchers in many research areas. For the purpose of maintaining stable running environment, system administrators and maintainers need to acquire the knowledge of events which happen in the environment to make quick responses for the encountered problems. This paper proposes the design of event stream processing and distributing system in CNGrid. This system provides the functionalities of gathering and classifying events produced by CNGrid, and distri-buting events to the related applications. The event factory module in the system performs the job of decoding events in multiple formats, filtering and preprocessing events, and finally packs them into a unified format to publish. This paper implements the basic functionalities of each part in the event stream processing and distributing system, and deploys it in CNGrid environment, then evaluates the delay resulted by the event factory module. The experiment results show that the delay by the event processing and distributing steps in the system is very low, and satisfying the requirement of low latency of sending events to related applications.

Key words: log processing, event stream, big-data analysis, high-performance computing