Rong-Tai Liu

A fast string-matching algorithm for network processor-based intrusion detection system

Network intrusion detection systems (NIDSs) are one of the latest developments in security. The matching of packet strings against collected signatures dominates signature-based NIDS performance. Network processors are also one of the fastest growing segments of the semiconductor market, because they are designed to provide scalable and flexible solutions that can accommodate change quickly and economically. This work presents a fast string-matching algorithm (called FNP) over the network processor platform that conducts matching sets of patterns in parallel. This design also supports numerous practical features such as case-sensitive string matching, signature prioritization, and multiple-content signatures. This efficient multiple-pattern matching algorithm utilizes the hardware facilities provided by typical network processors instead of employing the external lookup co-processors. To verify the efficiency and practicability of the proposed algorithm, it was implemented on the Vitesse IQ2000 network processor platform. The searching patterns used in the present experiments are derived from the well-known Snort ruleset cited by most open-source and commercial NIDSs. This work shows that combining our string-matching methodology, hashing engine supported by most network processors, and characteristics of current Snort signatures frequently improves performance and reduces number of memory accesses compared to conventional string-matching algorithms. Another contribution of this work is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multipattern matching algorithm performance.

A fast URL lookup engine for content-aware multi-gigabit switches

Cluster-based servers are one of the best solutions to build high-performance, scalable, and reliable Internet Web servers. A number of researches have been done about enabling the dispatcher in cluster-based Web servers to route the users requests based on higher layer information, such as URLs. Hashing functions and tree structure are often used to achieve the goal of URL lookup, but they may cause the problem of collision and result in unacceptable performance. This paper presents a fast scalable URL lookup mechanism that uses content addressable memory (CAM) as the basic hardware components. Our scheme not only supports exact matching of URL lookup, but also provides prefix-matching lookup ability so that it is very practically for URL content-filtering like systems. The proposed scheme takes constant time to lookup a URL and furnishes a rate of 100 million lookups per second. By applying the entry reuse concept, the expensive CAM space can be used in a more efficient way to store more URLs. With this fast URL lookup engine, the performance of content dispatchers or URL content filters can be greatly improved.

FTSE: the FNIP-like TCAM searching engine

As the Internet grows at a very rapid pace, so does the incidence of attack events and documented unlawful intrusions. The network intrusion detection systems (NIDSes) are designed to identify attacks against networks or a host that are invisible to firewalls, thus providing an additional layer of security. NIDSes detect and filter the malicious packets by inspecting packet payloads to find worm signatures. The payload inspection operation dominates the throughput of an NIDS since every byte of packet payload needs to be examined. At network speeds of 1 Gbps or above, it can be difficult to keep up with intrusion detection in software, and hardware systems or software with hardware assist are normally required. This paper presents FTSE, a ternary content addressable memory (TCAM) based pattern matching engine. In this paper we show how FTSE can be used effectively to perform string matching for thousands of strings at multiple-Gigabit speed. We also describe how FTSE can be implemented feasibly with an FPGA/ASIC, a 2.25 Mb TCAM, and a small SSRAM. Our analysis shows that this approach for string matching is very effective and the throughput of our design can achieve up to 8 Gbps for 2,085 snort rules.

A fast pattern matching algorithm for network processor-based intrusion detection system

Network intrusion detection systems (NIDS) monitor packets on the network and attempts to discover if a hacker is attempting to break into a system. The matching of packet strings against collected signatures dominates signature-based NIDS performance. Network processors are one of the fastest growing segments of the semiconductor market, because they are designed to provide scalable and flexible solutions that can accommodate change quickly and economically. This work presents a fast string matching algorithm (called FNP) over the network processor platform that conducts matching sets of patterns in parallel. FNP needs less number of memory accesses against conventional pattern-matching algorithms. Another contribution of this work is to highlight that, besides total number of searching patterns, shortest pattern length is also a major influence on NIDS multi-pattern matching algorithm performance.

A novel software-based MD5 checksum lookup scheme for anti-virus systems

In recent years, the size of virus signature databases has been growing rapidly, leading to a corresponding reduction in the performance of anti-virus (AV) software. In general, virus signature databases comprise string-based and hash-based (e.g., MD5) signatures. Currently the majority of signatures are hash-based and Cloud-based AV systems rely on them as the local cache to reduce the network loading. In this paper, we provide a novel scheme for looking up MD5 checksums to improve virus scanning performance involving hash-based signatures. The authors treat the range hash in which characters occur as a filter to avoid unnecessary lookups and keep the range of the exact search range to a minimum. The scheme is 135 times faster than ClamAVs in clean/general cases and only required 4MB of memory for hash-based filtering. This scheme could easily be extended to other hash-based applications.

Automatic NIDS Rule Generating System for Detecting HTTP-like Malware Communication

HTTP is the main protocol of the Internet and many network applications rely on it. Malware also utilizes it as a covert channel through which to evade the firewall (FW) or network intrusion detection system (NIDS). We recognize a malware, which employs HTTP to communicate as the HTTP-like Botnet. Some parts of the network traffic of an HTTP-like Botnet are different from normal HTTP applications. Based on the differences between HTTP-like Botnet traffic and normal HTTP applications, we developed an Automatic NIDS Rule Generating System (ARGS). The ARGS is a proof of concept (POC), which generates the corresponding NIDS rules efficiently and precisely from the input malign traffic (MT). ARGS is an incremental method to generate and optimize the rules. It can generate rules quickly and precisely without first requiring the collection of many malware samples for clustering. For practical purposes, we adopt Snort as our IDS engine in ARGS. In our experiments, the time required by ARGS to process MTs and generate corresponding rules is significantly shorter than existing solution when the rule-optimization is not required. Besides, the generated rule set can detect more 30% malware traffic compared to SourceFire IDS full-set and thus can efficiently stop the spreading of malware in time.

A predictive zero-day network defense using long-term port-scan recording

Zero-day attack is a critical network attack. The zero-day attack period (ZDAP) is the period from the release of malware/exploit until a patch becomes available. IDS/IPS cannot effectively block zero-day attacks because they use pattern-based signatures in general. This paper proposes a Prophetic Defender (PD) by which ZDAP can be minimized. Prior to actual attack, hackers scan networks to identify hosts with vulnerable ports. If this port scanning can be detected early, zero-day attacks will become detectable. PD architecture makes use of a honeypot-based pseudo server deployed to detect malicious port scans. A port-scanning honeypot was operated by us in 6 years from 2009 to 2015. By analyzing the 6-year port-scanning log data, we understand that PD is effective for detecting and blocking zero-day attacks. The block rate of the proposed architecture is 98.5%.

On the design of a cost effective network security switch architecture

This paper proposes a cost effective architecture for network security switch to deep inspect the traffic among switching ports. A security service engine (SSE) with packet deep inspection ability is also designed to accompany with manageable L2 switches. By properly configuring the VLAN parameters, packets from switch ports of the L2 switch are forwarded to the SSE, via the gigabit Ethernet interface, for deeply inspection. For security reason, abnormal/malicious packets are dropped by the SSE directly while normal packets are forwarded back to the switch to the correct output port. To evaluate the performance and latency of the proposed architecture, a cost effective P4-based SSE is also implemented as an intrusion detection and prevention system (IPS) with layer-7 content inspection function. The obtained measurements show that the proposed architecture is practical with high throughput and low latency. With IPC-based SSE implementation, the traditional L2 switches can now provide content security service in a very cost effective way.

A retargetable multiple string matching code generation for embedded network intrusion detection platforms

The common means of defense for network security systems is to block the intrusions by matching the signatures. Intrusion-signature matching is the critical operation. However, small and medium-sized enterprise (SME) or Small Office Home Office (SOHO) network security systems may not have sufficient resources to maintain good matching performance with full-set rules. Code generation is a technique used to convert data structures or instruction to other forms to obtain greater benefits within execution environments. This study analyzes intrusion detection system (IDS) signatures and discovers character occurrence to be significantly uneven. Based on this property, this study designs a method to generate a string matching source code according to the state table of AC algorithm for embedded network intrusion detection platforms. The generated source code requires less memory and relies not only on table lookup, but also on the ability of processor. This method can upgrade the performance by compiling optimization and contribute to the application of network processors and DSP-like based platforms. From evaluation, this method requires use of only 20% memory and can achieve 86% performance in clean traffic compared to the original Aho-Corasick algorithm (AC).

Fast proxyless stream-based anti-virus for Network Function Virtualization

Network anti-virus (AV) solutions are the first line of defense against malicious software. Traditional proxy-based network anti-virus solutions with store-scan-forward techniques decrease network performance and consume massive amounts of memory. Therefore, traditional solutions are not easily adaptable for Network Function Virtualization (NFV). This paper details the work on a novel virus scanning solution for NFV, called StreamAV. It does not require a proxy and maintains high network performance with less memory usage. StreamAV conducts policy matching on streams, rather than on complete files. This eliminates buffering, thereby accelerating traffic and requiring far less memory than solutions that scan complete files. The prototype was 40 times faster than its closest open source competitor, while its memory consumption was only a fraction of that of this competitor. Coverage was 100% with random test samples.