Abstract: Network traffic monitoring has been paid more and more attention in the telecom industry. As the network transmission bandwidth is increasing constantly， the traditional disk-based processing technology has not be able to meet the needs of operation management. Data stream processing technology could be applied to invent a new solution for the network traffic monitoring applications. SMART^[1，2] and RealMon^[3] are two network traffic monitoring tools developed by us， which are based on data stream processing technology. Aimming at meeting the requirements for network traffic monitoring of Shanghai Telecom’s backbone network， the design and application of the two systems are discussed. SMART is employing frequent items mining algorithm over data streams to support the Top-K queries over long-time sliding window. It achieves real-time traffic monitoring over lots of NetFlow streams. RealMon adapts dimension reduction analysis method to monitor the huge amount of SNMP messages which gather from routers in telecom backbone network. It monitors the correlations of thousands of network links and finds the traffic anomalies. Experiments and deployments in a real-life environment show that SMART and RealMon could meet the needs of traffic monitoring on backbone network and greatly improve the performance of the monitoring system.

Key words: data stream, network traffic monitoring, Top-K query, anomaly detection

摘要： 网络流量监测技术是电信运营商所关注的重点之一。随着网络传输带宽的不断增加，传统的基于磁盘的处理技术已经不能够满足运营管理的需要，数据流处理技术的出现为网络流量监测应用提供了一种新的解决方案。SMART^[1，2]和RealMon^[3]是将数据流处理技术应用于网络流量监测的两个应用系统。文中结合上海电信在骨干网流量监测上的应用需求，对这两个系统的设计与应用进行了介绍。SMART系统以数据流上的频繁项挖掘算法为基础，支持NetFlow数据流上长时间滑动窗口内的Top-K查询，实现了对网络流量分布的实时监测。RealMon系统采用数据流上的降维分析算法对SNMP数据流进行分析，通过监测网络链路上不同流量数据的相关性变化来发现异常流量。实验和在真实环境中的应用表明，SMART和RealMon系统能够满足骨干网流量监测应用的需要，并且极大地提高了监测系统的性能。

关键词: 数据流, 网络流量监测, Top-K查询, 异常检测