Yisen Wang

A less-greedy two-term Tsallis Entropy Information Metric approach for decision tree classification

Abstract The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. Most of them, however, are greedy algorithms that have the drawback of obtaining only local optimums. Besides, conventional split criteria they used, e.g. Shannon entropy, Gain Ratio and Gini index, are based on one-term that lack adaptability to different datasets. To address the above issues, we propose a less-greedy two-term Tsallis Entropy Information Metric (TEIM) algorithm with a new split criterion and a new construction method of decision trees. Firstly, the new split criterion is based on two-term Tsallis conditional entropy, which is better than conventional one-term split criteria. Secondly, the new tree construction is based on a two-stage approach that reduces the greediness and avoids local optimum to a certain extent. The TEIM algorithm takes advantages of the generalization ability of two-term Tsallis entropy and the low greediness property of two-stage approach. Experimental results on UCI datasets indicate that, compared with the state-of-the-art decision trees algorithms, the TEIM algorithm yields statistically significantly better decision trees and is more robust to noise.

Link sign prediction by Variational Bayesian Probabilistic Matrix Factorization with Student-t Prior

Abstract In signed social networks, link sign prediction refers to using the observed link signs to infer the signs of the remaining links, which is important for mining and analyzing the evolution of social networks. The widely used matrix factorization-based approach – Bayesian Probabilistic Matrix Factorization (BMF), assumes that the noise between the real and predicted entry is Gaussian noise, and the prior of latent features is multivariate Gaussian distribution. However, Gaussian noise model is sensitive to outliers and is not robust. Gaussian prior model neglects the differences between latent features, that is, it does not distinguish between important and non-important features. Thus, Gaussian assumption based models perform poorly on real-world (sparse) datasets. To address these issues, a novel Variational Bayesian Probabilistic Matrix Factorization with Student-t prior model (TBMF) is proposed in this paper. A univariate Student-t distribution is used to fit the prediction noise, and a multivariate Student-t distribution is adopted for the prior of latent features. Due to the high kurtosis of Student-t distribution, TBMF can select informative latent features automatically, characterize long-tail cases and obtain reasonable representations on many real-world datasets. Experimental results show that TBMF improves the prediction performance significantly compared with the state-of-the-art algorithms, especially when the observed links are few.

Unifying attribute splitting criteria of decision trees by Tsallis entropy

Owing to its simplicity and flexibility, the decision tree remains an important analysis tool in many real-world learning tasks. A lot of decision tree algorithms have been proposed, such as ID3, C4.5 and CART, which represent three most prevalent criteria of attribute splitting, i.e., Shannon entropy, Gain Ratio and Gini index respectively. These splitting criteria seem to be independent and to work in isolation. However, in this paper, we find that these three attribute splitting criteria can be unified in a Tsallis entropy framework. More importantly, theoretically, we reveal the relations between Tsallis entropy and the above three prevalent attribute splitting criteria. In addition, we generalize the splitting criterion of the decision tree, and provide a new simple but efficient approach, Unified Tsallis Criterion Decision Tree algorithm (UTCDT), to enhance the performance of the decision tree. Experimental evidences demonstrate that UTCDT achieves statistically significant improvement over the classical decision tree algorithms, even yields comparable performance to state-of-the-art classification algorithm.

Collaborative Representation Cascade for Single-Image Super-Resolution

Most recent learning-based single-image super-resolution methods first interpolate the low-resolution (LR) input, from which overlapped LR features are then extracted to reconstruct their high-resolution (HR) counterparts and the final HR image. However, most of them neglect to take advantage of the intermediate recovered HR image to enhance image quality further. We conduct principal component analysis (PCA) to reduce LR feature dimension. Then we find that the number of principal components after conducting PCA in the LR feature space from the reconstructed images is larger than that from the interpolated images by using bicubic interpolation. Based on this observation, we present an unsophisticated yet effective framework named collaborative representation cascade (CRC) that learns multilayer mapping models between LR and HR feature pairs. In particular, we extract the features from the intermediate recovered image to upscale and enhance LR input progressively. In the learning phase, for each cascade layer, we use the intermediate recovered results and their original HR counterparts to learn single-layer mapping model. Then, we use this single-layer mapping model to super-resolve the original LR inputs. And the intermediate HR outputs are regarded as training inputs for the next cascade layer, until we obtain multilayer mapping models. In the reconstruction phase, we extract multiple sets of LR features from the LR image and intermediate recovered. Then, in each cascade layer, mapping model is utilized to pursue HR image. Our experiments on several commonly used image SR testing datasets show that our proposed CRC method achieves state-of-the-art image SR results, and CRC can also be served as a general image enhancement framework.

A novel feature subspace selection method in random forests for high dimensional data.

Random forests are a class of ensemble methods for classification and regression with randomizing mechanism in bagging instances and selecting feature subspace. For high dimensional data, the performance of random forests degenerates because of the random sampling feature subspace for each node in the construction of decision trees. To address the issue, in this paper, we propose a new Principal Component Analysis and Stratified Sampling based method, called PCA-SS, for feature subspace selection in random forests with high dimensional data. For each decision tree in the forests, we firstly create the training data by bagging instances and partition the feature set into several feature subsets. Principal Component Analysis (PCA) is applied on each feature subset to obtain transformed features. All the principal components are retained in order to preserve the variability information of the data. Secondly, depending on a certain principal components principle, the transformed features are partitioned into informative and less informative parts. When constructing each node of decision trees, a feature subspace is selected by stratified sampling method from the two parts. The PCA-SS based Random Forests algorithm, named PSRF, ensures enough informative features for each tree node, and it also increases the diversity between the trees to a certain extent. Experimental results demonstrate that the proposed PSRF significantly improves the performance of random forests when dealing with high dimensional data, compared with the state-of-the-art random forests algorithms.

End-to-end coding for TCP

Although widely used, TCP has many limitations in meeting the throughput and latency requirements of applications in wireless networks, high-speed data center networks, and heterogeneous multi-path networks. Instead of relying purely on retransmission upon packet loss, coding has potential to improve the performance of TCP by ensuring better transmission reliability. Coding has been verified to work well at the link layer but has not been fully studied at the transport layer. There are many advantages but also challenges in exploiting coding at the transport layer. In this article, we focus on how to leverage end-to-end coding in TCP. We reveal the problems TCP faces and the opportunities coding can bring to improve TCP performance. We further analyze the challenges faced when applying the coding techniques to TCP and present the current applications of coding in TCP.

A generic denoising framework via guided principal component analysis

Though existing state-of-the-art denoising algorithms, such as BM3D, LPG-PCA and DDF, obtain remarkable results, these methods are not good at preserving details at high noise levels, sometimes even introducing non-existent artifacts. To improve the performance of these denoising methods at high noise levels, a generic denoising framework is proposed in this paper, which is based on guided principle component analysis (GPCA). The propose framework can be split into two stages. First, we use statistic test to generate an initial denoised image through back projection, where the statistical test can detect the significantly relevant information between the denoised image and the corresponding residual image. Second, similar image patches are collected to form different patch groups, and local basis are learned from each patch group by principle component analysis. Experimental results on natural images, contaminated with Gaussian and non-Gaussian noise, verify the effectiveness of the proposed framework.

A Novel Consistent Random Forest Framework: Bernoulli Random Forests

Random forests (RFs) are recognized as one type of ensemble learning method and are effective for the most classification and regression tasks. Despite their impressive empirical performance, the theory of RFs has yet been fully proved. Several theoretically guaranteed RF variants have been presented, but their poor practical performance has been criticized. In this paper, a novel RF framework is proposed, named Bernoulli RFs (BRFs), with the aim of solving the RF dilemma between theoretical consistency and empirical performance. BRF uses two independent Bernoulli distributions to simplify the tree construction, in contrast to the RFs proposed by Breiman. The two Bernoulli distributions are separately used to control the splitting feature and splitting point selection processes of tree construction. Consequently, theoretical consistency is ensured in BRF, i.e., the convergence of learning performance to optimum will be guaranteed when infinite data are given. Importantly, our proposed BRF is consistent for both classification and regression. The best empirical performance is achieved by BRF when it is compared with state-of-the-art theoretical/consistent RFs. This advance in RF research toward closing the gap between theory and practice is verified by the theoretical and experimental studies in this paper.

Improving decision trees by Tsallis Entropy Information Metric method

The construction of efficient and effective decision trees remains a key topic in machine learning because of their simplicity and flexibility. A lot of heuristic algorithms have been proposed to construct near-optimal decision trees. Most of them, however, are greedy algorithms that have the drawback of obtaining only local optimums. Besides, conventional split criteria they used, e.g. Shannon entropy, Gain Ratio and Gini index, cannot select informative attributes efficiently. To address the above issues, we propose a novel Tsallis Entropy Information Metric (TEIM) algorithm with a new split criterion and a new construction method of decision trees. Firstly, the new split criterion is based on two terms of Tsallis conditional entropy, which is better than conventional split criteria. Secondly, the new construction method is based on a two-stage approach that avoids local optimum to a certain extent. The TEIM algorithm takes advantages of the generalization ability of Tsallis entropy and the low greediness property of two-stage approach. Experimental results on UCI datasets indicate that, compared with the state-of-the-art decision trees algorithms, the TEIM algorithm yields statistically significantly better decision trees in classification accuracy as well as tree complexity.

Robust Survey Aggregation with Student-t Distribution and Sparse Representation

Most existing survey aggregation methods assume that the sample data follow Gaussian distribution. However, these methods are sensitive to outliers, due to the thin-tailed property of Gaussian distribution. To address this issue, we propose a robust survey aggregation method based on Student-t distribution and sparse representation. Specifically, we assume that the samples follow Student-t distribution, instead of the common Gaussian distribution. Due to the Student-t distribution, our method is robust to outliers, which can be explained from both Bayesian point of view and non-Bayesian point of view. In addition, inspired by James-Stain estimator (JS) and Compressive Averaging (CAvg), we propose to sparsely represent the global mean vector by an adaptive basis comprising both dataspecific basis and combined generic basis. Theoretically, we prove that JS and CAvg are special cases of our method. Extensive experiments demonstrate that our proposed method achieves significant improvement over the state-of-the-art methods on both synthetic and real datasets.