关键词: 数据流, 网络流量监测, Top-K查询, 异常检测

Abstract: Network traffic monitoring has been paid more and more attention in the telecom industry. As the network transmission bandwidth is increasing constantly， the traditional disk-based processing technology has not be able to meet the needs of operation management. Data stream processing technology could be applied to invent a new solution for the network traffic monitoring applications. SMART^[1，2] and RealMon^[3] are two network traffic monitoring tools developed by us， which are based on data stream processing technology. Aimming at meeting the requirements for network traffic monitoring of Shanghai Telecom’s backbone network， the design and application of the two systems are discussed. SMART is employing frequent items mining algorithm over data streams to support the Top-K queries over long-time sliding window. It achieves real-time traffic monitoring over lots of NetFlow streams. RealMon adapts dimension reduction analysis method to monitor the huge amount of SNMP messages which gather from routers in telecom backbone network. It monitors the correlations of thousands of network links and finds the traffic anomalies. Experiments and deployments in a real-life environment show that SMART and RealMon could meet the needs of traffic monitoring on backbone network and greatly improve the performance of the monitoring system.

Key words: data stream, network traffic monitoring, Top-K query, anomaly detection