Design, Implementation, and Evaluation of Network Monitoring Tasks with the STREAM Data Stream Management System

Abstract

The heavy load and rich variety of data on the Internet has resulted in the need to gain an understanding of the characteristics of the traffic to better plan, develop, and implement new network devices, applications, and protocols. In order to obtain such knowledge, network monitoring is becoming more and more important. However, tools available for network monitoring are restricted to either offline analysis in DBMSs or online analysis through hard-coded continuous queries. Many streaming applications would benefit from a system where network monitoring queries effectively can be inserted, deleted, modified, and processed online in a continuous and real time manner. Data Stream Management System (DSMS) is a promising technology with respect to the needs of network monitoring, because it is designed to meet the above requirements generated by many streaming applications. In the present paper, an experimental analysis of STREAM as a network monitoring tool is performed. STREAM is a general-purpose DSMS and its continuous query language is known as CQL (Continuous Query Language). We investigate whether the current implementation of CQL operators provides us the possibility of expressing a wide-ranged set of network monitoring queries. Furthermore, STREAM's performance is measures by accomplishing several experiments that are processed online over real network traffic. Results reveal that STREAM can handle network loads up to 30 Mb/s for simple queries, and up to approximately 3 Mb/s complex queries. When queries are executed concurrently, STREAM can handle network loads up to approximately 2.5 Mb/s, strongly depending on the complexity and number of queries. STREAM provides a well-sized set of operators that provides us the possibility of expressing many types of queries. However, network monitoring queries are restricted by lack of specific network data types and operators. Consequently, these queries are expressed in cumbersome ways. STREAM manages to process network monitoring queries online in a continuous manner, nevertheless at a very limited network load. Thus, the applicability of STREAM as a network monitoring tool is restricted.