Jungpin Wu

Using Data Mining to Study Upstream and Downstream Causal Relationship in Stock Market

To understand the causal relationship of stock market is always a top priority for investors. Most investors use some fundamental knowledge and basic analysis techniques to analyze or predict the trends. However, there are always some other factors beyond our control or unexpected events that might affect the stock market one way or the other. After working on data mining with good results, we found inter-transaction mining can help answer the above questions in a systemic way. Our experiments show that causal relationship between upstream and downstream stocks do exist. To simplify our discussion, we focus on the electrical industrial stocks.

Effective Application of Improved Profit-Mining Algorithm for the Interday Trading Model

Many real world applications of association rule mining from large databases help users make better decisions. However, they do not work well in financial markets at this time. In addition to a high profit, an investor also looks for a low risk trading with a better rate of winning. The traditional approach of using minimum confidence and support thresholds needs to be changed. Based on an interday model of trading, we proposed effective profit-mining algorithms which provide investors with profit rules including information about profit, risk, and winning rate. Since profit-mining in the financial market is still in its infant stage, it is important to detail the inner working of mining algorithms and illustrate the best way to apply them. In this paper we go into details of our improved profit-mining algorithm and showcase effective applications with experiments using real world trading data. The results show that our approach is practical and effective with good performance for various datasets.

Efficient Mining of Frequent Closed Itemsets without Closure Checking

Most existing algorithms for mining frequent closed itemsets have to check whether a newly generated itemset is a frequent closed itemset by using the subset checking technique. To do this, a storing structure is required to keep all known frequent itemsets and candidates. It takes additional processing time and memory space for closure checking. To remedy this problem, an efficient approach called closed itemset mining with no closure checking algorithm is proposed. We use the information recorded in a FP-tree to identify the items that will not constitute closed itemsets. Using this information, we can generate frequent closed itemsets directly. It is no longer necessary to check whether an itemset is closed or not when it is generated. We have implemented our algorithm and made many performance experiments. The results show that our approach has better performance in the runtime and memory space utilization. Moreover, this approach is also suitable for parallel mining of frequent closed itemsets.

Discovering long maximal frequent pattern

Association rule mining, the most commonly used method for data mining, has numerous applications. Although many approaches that can find association rules have been developed, most utilize maximum frequent itemsets that are short. Existing methods fail to perform well in applications involving large amounts of data and incur longer itemsets. Apriori-like algorithms have this problem because they generate many candidate itemsets and spend considerable time scanning databases; that is, their processing method is bottom-up and layered. This paper solves this problem via a novel hybrid Multilevel-Search algorithm. The algorithm concurrently uses the bidirectional Pincer-Search and parameter prediction mechanism along with the bottom-up search of the Parameterised method to reduce the number of candidate itemsets and consequently, the number of database scans. Experimental results demonstrate that the proposed algorithm performs well, especially when the length of the maximum frequent itemsets are longer than or equal to eight. The concurrent approach of our multilevel algorithm results in faster execution time and improved efficiency.

Efficient Mining of Profit Rules from Closed Inter-Transaction Itemsets

Data mining applications in financial sectors are very common since investors can apply the resultant rules to make profits. Profit mining algorithms in particular, such as PRMiner, can generate profit rules that meet the expectations of investors regarding profit, risk, and win rate. However, most of such algorithms are not efficient due to the long processing time involving going through the whole search space in complex dynamical systems of financial markets. Hence, we propose a new approach in this paper to solve the problem by using closed itemsets to obtain profit rules without processing the entire trading rules. Based on the inter-day modeling, we analyze inter-transactions and conduct trading simulations to predict trading results for efficient profit rule generation. We develop two algorithms of JCMiner and ATMiner to process closed itemsets, which have better performance than the approach of PRMiner, especially for the large number of itemsets and large datasets. According to the experimental results, our algorithms outperform PRMiner in various experimental scenarios, i.e., mining parameters, the number of items in a transaction, and the number of transactions in a dataset.

Fast and Effective Generation of Candidate-Sequences for Sequential Pattern Mining

The existing sequential pattern mining algorithms fall into two categories. One is the candidate-generation-and-test approach such as GSP, and the other is the pattern-growth approach such as PrefixSpan. Both GSP and PrefixSpan require setting the minimum support before their execution. We propose a new approach, called Fast and Effective Generation of Candidate-sequences (FEGC), to mine sequential patterns without predetermining the minimum support threshold. The main contribution is to scan all transactions in the database once and generate all the subsequences with their support counters. The experiments show that our algorithm performs well in various datasets.

A flexible and efficient sequential pattern mining algorithm

Sequential pattern mining has gathered great attention in recent years due to its broad applications. Most of the existing methods are in two categories: 1) candidate-generation-and-test approaches such as GSP, requiring multiple database scans, 2) pattern-growth approaches such as PrefixSpan, scanning the projected database which may be several times larger than the original database. Methods from both categories must set minimum support thresholds in advance. To remedy the problems, we propose a new approach, Fast Sequential Pattern Enumeration (FSPE), to mine sequential patterns without the need to predetermine the minimum support threshold. The FSPE scans the transaction database only once to enumerate all candidate sequences with efficient indexing of their support counters. Using our approach one can easily produce meaningful rules for any item that appears at least once in the sequence database.

An Efficient Sequential Pattern Mining Algorithm Based on the 2-Sequence Matrix

Sequential pattern mining has become more and more popular in recent years due to its wide applications and the fact that it can find more information than association rules. Two famous algorithms in sequential pattern mining are AprioriAll and PrefixSpan. These two algorithms not only need to scan a database or projected-databases many times, but also require setting a minimal support threshold to prune infrequent data to obtain useful sequential patterns efficiently. In addition, they must rescan the database if new items or sequences are added. In this paper, we propose a novel algorithm called efficient sequential pattern enumeration (ESPE) to solve the above problems. In addition, our method can be applied in many applications, such as for the itemsets appearing at the same time in a sequence. In our experiments, we show that the performance of ESPE is better than the other two methods using various datasets.

A Framework to Evaluate the Advertisement of Digital Signage over the Internet

With advances in multimedia and Internet technologies, advertising has entered a digital network era. Digital signage on the Internet has replaced traditional static billboards and becomes the best choice for advertising media. Therefore, verifying the effectiveness of advertisements is critical for advertising agencies to make better decisions in order to increase sales for their clients. This paper proposes an evaluation framework to measure the effectiveness of advertisements on digital signage over the Internet. The framework combines face recognition and data mining techniques to detect in real-time whether customers are looking at digital signage and to record when potential customers are paying attention to advertisements. The longer the total duration is, the better the effect can be assessed. The collected data can be used to evaluate the effectiveness of advertisements and promote targeted sales. Our experiments show that our approach is feasible and meets user expectations.