National Technical Reports Library

The National Technical Information Service acquires, indexes, abstracts, and archives the largest collection of U.S. government-sponsored technical reports in existence. The NTRL offers online, free and open access to these authenticated government technical reports. Technical reports and documents in its repository may be available online for free either from the issuing federal agency, the U.S. Government Publishing Office’s Federal Digital System website, or through search engines.

Details

Efficient Analysis of Live and Historical Streaming Data and Its Application to Cybersecurity.

DE2008920351

Publication Date	2007
Personal Author	Reiss, F.; Wu, K.; Stockinger, K.; Hellerstein, J. M.; Shoshani, A.
Page Count	21
Abstract	Applications that query data streams in order to identify trends, patterns, or anomalies can often benefit from comparing the livestream data with archived historical stream data. However, searching this historical data in real time has been considered so far to be prohibitively expensive. One of the main bottlenecks is the update costs of the indices over the archived data. In this paper, we address this problem by using our highly-efficient bitmap indexing technology (called FastBit) and demonstrate that the index update operations are sufficiently efficient for this bottleneck to be removed. We describe our prototype system based on the TelegraphCQ streaming query processor and the FastBit bitmap index. We present a detailed performance evaluation of our system using a complex query workload for analyzing real network traffic data. The combined system uses TelegraphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We demonstrate that our system can simultaneously analyze (1) live streams with high data rates and (2) a large repository of historical stream data.
Keywords	Cybersecurity Stream query Archival data Efficient query processing FastBit Bitmap indexing Homeland security Indices update costs TelegraphCQ Network traffic analysis
Source Agency	Technical Information Center Oak Ridge Tennessee
Corporate Authors	Lawrence Berkeley National Lab., CA.; California Univ., Berkeley. Computer Science Div.; Department of Energy, Washington, DC.
Supplemental Notes	Prepared in cooperation with California Univ., Berkeley. Computer Science Div. Sponsored by Department of Energy, Washington, DC.
Document Type	Technical Report
NTIS Issue Number	200817
Contract Number	DE-AC02-05CH11231

Efficient Analysis of Live and Historical Streaming Data and Its Application to Cybersecurity.

Efficient Analysis of Live and Historical Streaming Data and Its Application to Cybersecurity.
DE2008920351

Publication Date	2007
Personal Author	Reiss, F.; Wu, K.; Stockinger, K.; Hellerstein, J. M.; Shoshani, A.
Page Count	21
Abstract	Applications that query data streams in order to identify trends, patterns, or anomalies can often benefit from comparing the livestream data with archived historical stream data. However, searching this historical data in real time has been considered so far to be prohibitively expensive. One of the main bottlenecks is the update costs of the indices over the archived data. In this paper, we address this problem by using our highly-efficient bitmap indexing technology (called FastBit) and demonstrate that the index update operations are sufficiently efficient for this bottleneck to be removed. We describe our prototype system based on the TelegraphCQ streaming query processor and the FastBit bitmap index. We present a detailed performance evaluation of our system using a complex query workload for analyzing real network traffic data. The combined system uses TelegraphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We demonstrate that our system can simultaneously analyze (1) live streams with high data rates and (2) a large repository of historical stream data.
Keywords	Cybersecurity Stream query Archival data Efficient query processing FastBit Bitmap indexing Homeland security Indices update costs TelegraphCQ Network traffic analysis
Source Agency	Technical Information Center Oak Ridge Tennessee
Corporate Authors	Lawrence Berkeley National Lab., CA.; California Univ., Berkeley. Computer Science Div.; Department of Energy, Washington, DC.
Supplemental Notes	Prepared in cooperation with California Univ., Berkeley. Computer Science Div. Sponsored by Department of Energy, Washington, DC.
Document Type	Technical Report
NTIS Issue Number	200817
Contract Number	DE-AC02-05CH11231