Yunlong Feng

Unified approach to coefficient-based regularized regression

Abstract In this paper, we consider the coefficient-based regularized least-squares regression problem with the l q -regularizer ( 1 ≤ q ≤ 2 ) and data dependent hypothesis spaces. Algorithms in data dependent hypothesis spaces perform well with the property of flexibility. We conduct a unified error analysis by a stepping stone technique. An empirical covering number technique is also employed in our study to improve sample error. Comparing with existing results, we make a few improvements: First, we obtain a significantly sharper learning rate that can be arbitrarily close to O ( m − 1 ) under reasonable conditions, which is regarded as the best learning rate in learning theory. Second, our results cover the case q = 1 , which is novel. Finally, our results hold under very general conditions.

Kernelized elastic net regularization: Generalization bounds, and sparse recovery

Kernelized elastic net regularization (KENReg) is a kernelization of the well-known elastic net regularization (Zou & Hastie, 2005). The kernel in KENReg is not required to be a Mercer kernel since it learns from a kernelized dictionary in the coefficient space. Feng, Yang, Zhao, Lv, and Suykens (2014) showed that KENReg has some nice properties including stability, sparseness, and generalization. In this letter, we continue our study on KENReg by conducting a refined learning theory analysis. This letter makes the following three main contributions. First, we present refined error analysis on the generalization performance of KENReg. The main difficulty of analyzing the generalization error of KENReg lies in characterizing the population version of its empirical target function. We overcome this by introducing a weighted Banach space associated with the elastic net regularization. We are then able to conduct elaborated learning theory analysis and obtain fast convergence rates under proper complexity and regularity assumptions. Second, we study the sparse recovery problem in KENReg with fixed design and show that the kernelization may improve the sparse recovery ability compared to the classical elastic net regularization. Finally, we discuss the interplay among different properties of KENReg that include sparseness, stability, and generalization. We show that the stability of KENReg leads to generalization, and its sparseness confidence can be derived from generalization. Moreover, KENReg is stable and can be simultaneously sparse, which makes it attractive theoretically and practically.

Robust Low-Rank Tensor Recovery With Regularized Redescending M-Estimator

This paper addresses the robust low-rank tensor recovery problems. Tensor recovery aims at reconstructing a low-rank tensor from some linear measurements, which finds applications in image processing, pattern recognition, multitask learning, and so on. In real-world applications, data might be contaminated by sparse gross errors. However, the existing approaches may not be very robust to outliers. To resolve this problem, this paper proposes approaches based on the regularized redescending M-estimators, which have been introduced in robust statistics. The robustness of the proposed approaches is achieved by the regularized redescending M-estimators. However, the nonconvexity also leads to a computational difficulty. To handle this problem, we develop algorithms based on proximal and linearized block coordinate descent methods. By explicitly deriving the Lipschitz constant of the gradient of the data-fitting risk, the descent property of the algorithms is present. Moreover, we verify that the objective functions of the proposed approaches satisfy the Kurdyka-Łojasiewicz property, which establishes the global convergence of the algorithms. The numerical experiments on synthetic data as well as real data verify that our approaches are robust in the presence of outliers and still effective in the absence of outliers.

A Rank-One Tensor Updating Algorithm for Tensor Completion

In this letter, we propose a rank-one tensor updating algorithm for solving tensor completion problems. Unlike the existing methods which penalize the tensor by using the sum of nuclear norms of unfolding matrices, our optimization model directly employs the tensor nuclear norm which is studied recently. Under the framework of the conditional gradient method, we show that at each iteration, solving the proposed model amounts to computing the tensor spectral norm and the related rank-one tensor. Because the problem of finding the related rank-one tensor is NP-hard, we propose a subroutine to solve it approximately, which is of low computational complexity. Experimental results on real datasets show that our algorithm is efficient and effective.

Rank-1 Tensor Properties with Applications to a Class of Tensor Optimization Problems

This paper studies models and algorithms for a class of tensor optimization problems, based on a rank-1 equivalence property between a tensor and certain unfoldings. It is first shown that in

Least-squares regularized regression with dependent samples and q-penalty

d

Robust support vector machines for classification with nonconvex and smooth losses

th order tensor space, the set of rank-1 tensors is the same as the intersection of

Learning theory estimates with observations from general stationary stochastic processes

\lceil \log_2(d) \rceil

Robust Gradient Learning With Applications

tensor sets, of which tensors have a specific rank-1 balanced unfolding matrix. Moreover, the number

Learning performance of elastic-net regularization

\lceil \log_2(d) \rceil