Yuexiang Yang

Analyzing Malware by Abstracting the Frequent Itemsets in API Call Sequences

Analyzing the usage of Windows Application Program Interface (API) is a common way to understand behaviors of Malicious Software (malware) in either static analysis or dynamic analysis methods. In this work, we focus on the usage of frequent messages in API call sequences, and we hypothesize that frequent itemsets composed of API names and/or API arguments could be valuable in the identification of the behavior of malware. For verification, we introduced clustering processes of malware binaries based on their frequent itemsets of API call sequences, and we evaluated the performance of malware clustering. Specific implementation processes for malware clustering, including API calls abstraction, frequent itemsets mining and similarity calculation, are illustrated. The experiment upon a big malware dataset demonstrated that merely using the frequent messages of API call sequences can achieve a high precision for malware clustering while significantly reducing the computation time. This also proves the importance of frequent itemsets in API call sequences for identifying the behavior of malware.

CBM: Free, Automatic Malware Analysis Framework Using API Call Sequences

Classic static code analysis for malware is ineffective when challenged by diverse variants. As a result, dynamic analysis based on malware behavior is becoming thriving in malware research. Most current dynamic analysis systems are provided as online services for common users. However, it is inconvenient and ineffective to use online services for the analysis of a big malware dataset. In this paper, we propose a framework named CBM enabling tailored construction of an automated system for malware analysis. In CBM, API call sequences are extracted as malware behavior reports by dynamic behavior analysis tool, and then API calls will be transformed to byte-based sequential data for further analysis by a novel malware behavior representation called BBIS. The peculiar characteristic of CBM is that it can be customized freely, contrary to current online systems, which supports local deployment and runs mass malware analysis automatically. Experiments were carried out on a large-scale malware dataset, which have demonstrated that CBM is more efficient in reducing storage size and computation cost while keeping a high precision for malware clustering.

A Novel Hybrid Mobile Malware Detection System Integrating Anomaly Detection With Misuse Detection

As the dominator of the Smartphone operating system market, Android has attracted the attention of malware authors and researchers alike. The number of Android malware is increasing rapidly regardless of the considerable number of proposed malware analysis systems. In this paper, by taking advantages of low false-positive rate of misuse detection and the ability of anomaly detection to detect zero-day malware, we propose a novel hybrid detection system based on a new open-source framework CuckooDroid, which enables the use of Cuckoo Sandboxs features to analyze Android malware through dynamic and static analysis. Our proposed system mainly consist of two parts, a misuse detector performing known malware detection and classification through combining static analysis with dynamic analysis; an anomaly detector performing abnormal apps detection through dynamic analysis. We evaluate our method with 5560 malware samples and 12000 benign samples. Experiments shows that our misuse detector with hybrid analysis can accurately detect and classify malware samples with an average positive rate 98.79%, 98.32% respectively; it is worth noting that our anomaly detector by dynamic analysis is capable of detecting zero-day malware with a low false negative rate (1.24%) and acceptable false positive rate (2.24%). Our proposed detection system is mainly designed for App store markets and the ordinary users who can access our system through mobile cloud service.

Detecting P2P bots by mining the regional periodicity

Peer-to-peer (P2P) botnets outperform the traditional Internet relay chat (IRC) botnets in evading detection and they have become a prevailing type of threat to the Internet nowadays. Current methods for detecting P2P botnets, such as similarity analysis of network behavior and machine-learning based classification, cannot handle the challenges brought about by different network scenarios and botnet variants. We noticed that one important but neglected characteristic of P2P bots is that they periodically send requests to update their peer lists or receive commands from botmasters in the command-and-control (C&C) phase. In this paper, we propose a novel detection model named detection by mining regional periodicity (DMRP), including capturing the event time series, mining the hidden periodicity of host behaviors, and evaluating the mined periodic patterns to identify P2P bot traffic. As our detection model is built based on the basic properties of P2P protocols, it is difficult for P2P bots to avoid being detected as long as P2P protocols are employed in their C&C. For hidden periodicity mining, we introduce the so-called regional periodic pattern mining in a time series and present our algorithms to solve the mining problem. The experimental evaluation on public datasets demonstrates that the algorithms are promising for efficient P2P bot detection in the C&C phase.

An Improved Method for Multi-class Support Vector Machines

Based on analyzing the advantages and disadvantages of existing multi-class support vector machines, we construct an improved multi-class support vector machines based on binary tree structure, adopting a new metrics to determine the classification order which determines each sub-classifier and its location, the new metrics synthesizes mixed degree and distance between classes. Then we do a measuring experiment using the improved multi-class support vector machines, which identifies five major P2P IPTV applications, the results show that our method is better than one-against-all and one-against-one method.

A Classification Method of Unstructured P2P Multicast Video Streaming Based on SVM

The classification of unstructured P2P multicast video streaming is the premise for playing online linkage and real-time evidence in the process of network monitoring management. A new classification method is demonstrated, and some real-time protocol behavior features are figured in this paper, which is found out through distinguishing packet type and transmission direction. With these accessible features a multi-classification support vector machine is built to identify applications of unstructured P2P multicast video streaming. The result of experiment shows that the method has high identification accuracy and satisfying real-time characteristic.

Hardware assisted hypervisor introspection

In this paper, we introduce hypervisor introspection, an out-of-box way to monitor the execution of hypervisors. Similar to virtual machine introspection which has been proposed to protect virtual machines in an out-of-box way over the past decade, hypervisor introspection can be used to protect hypervisors which are the basis of cloud security. Virtual machine introspection tools are usually deployed either in hypervisor or in privileged virtual machines, which might also be compromised. By utilizing hardware support including nested virtualization, EPT protection and #BP, we are able to monitor all hypercalls belongs to the virtual machines of one hypervisor, include that of privileged virtual machine and even when the hypervisor is compromised. What’s more, hypercall injection method is used to simulate hypercall-based attacks and evaluate the performance of our method. Experiment results show that our method can effectively detect hypercall-based attacks with some performance cost. Lastly, we discuss our furture approaches of reducing the performance cost and preventing the compromised hypervisor from detecting the existence of our introspector, in addition with some new scenarios to apply our hypervisor introspection system.

Fine-grained P2P traffic classification by simply counting flows

The continuous emerging of peer-to-peer (P2P) applications enriches resource sharing by networks, but it also brings about many challenges to network management. Therefore, P2P applications monitoring, in particular, P2P traffic classification, is becoming increasingly important. In this paper, we propose a novel approach for accurate P2P traffic classification at a fine-grained level. Our approach relies only on counting some special flows that are appearing frequently and steadily in the traffic generated by specific P2P applications. In contrast to existing methods, the main contribution of our approach can be summarized as the following two aspects. Firstly, it can achieve a high classification accuracy by exploiting only several generic properties of flows rather than complicated features and sophisticated techniques. Secondly, it can work well even if the classification target is running with other high bandwidth-consuming applications, outperforming most existing host-based approaches, which are incapable of dealing with this situation. We evaluated the performance of our approach on a real-world trace. Experimental results show that P2P applications can be classified with a true positive rate higher than 97.22% and a false positive rate lower than 2.78%.

DroidContext: Identifying Malicious Mobile Privacy Leak Using Context

Serious concerns have been raised about stealthy leakage of users privacy in mobile apps, and many recent approaches are also proposed to detect privacy leak in these apps. However, more and more benign mobile apps have to send out users privacy for legitimate functions or user intention. To evade detection, new mobile malware starts to mimic privacy-related behaviors of benign apps that provide similar functionality, and mix malicious privacy leak with benign ones to reduce the chance of being observed. Since prior proposed approaches primarily focus on the privacy leak discovery, these evasive techniques in new mobile malware will make differentiating between malicious and benign privacy disclosures a difficult task during privacy leak analysis. In this paper, we propose DroidContext, an automated system that detects truly malicious privacy leakages in Android apps. DroidContext differentiates malicious and benign privacy disclosures using contexts (e.g., activation events and dependent operations that trigger and control privacy leak execution), purifying the privacy leak detection results for automatic and easy interpretation by filtering out benign privacy disclosures. We implement a prototype of DroidContext and evaluate DroidContext on 5560 mobile malware and 4800 Apkure apps. Experiment results show that, on average, DroidContext achieves a high 92.85% true positive during malicious privacy identification and the 95.45% true positive during benign privacy disclosures identification. The necessity of proposed contexts is also evaluated. Evaluation indicates that to keep the accuracy of privacy disclosure classification, our proposed contexts are all necessary.

SPEMS: A Stealthy and Practical Execution Monitoring System Based on VMI

Dynamic analyzing has been proposed for over decades to tracing the execution of programs. However, most of them need an agent installed inside the execution environment, which is easy to be detected and bypassed. To solve the problem, we proposed a system named SPEMS which utilized virtual machine introspection (VMI) technology to stealthily monitor the execution of programs inside virtual machines. SPEMS integrates and improves multiple open-source software tools. By inspecting the whole process of sample preparation, execution tracing and analysis, it is able to be applied in large scale program monitoring, malware analyzing and memory forensics. Experiments results show our system has remarkable performance improvement compared with former works.