Vitaly Maiorov

Approximation bounds for smooth functions in C(R/sup d/) by neural and mixture networks

We consider the approximation of smooth multivariate functions in C(IRd) by feedforward neural networks with a single hidden layer of nonlinear ridge functions. Under certain assumptions on the smoothness of the functions being approximated and on the activation functions in the neural network, we present upper bounds on the degree of approximation achieved over the domain IRd, thereby generalizing available results for compact domains. We extend the approximation results to the so-called mixture of expert architecture, which has received considerable attention in recent years, showing that the same type of approximation bound may be achieved.

Lower bounds for approximation by MLP neural networks

Abstract The degree of approximation by a single hidden layer MLP model with n units in the hidden layer is bounded below by the degree of approximation by a linear combination of n ridge functions. We prove that there exists an analytic, strictly monotone, sigmoidal activation function for which this lower bound is essentially attained. We also prove, using this same activation function, that one can approximate arbitrarily well any continuous function on any compact domain by a two hidden layer MLP using a fixed finite number of units in each layer.

Almost Linear VC Dimension Bounds for Piecewise Polynomial Networks

We compute upper and lower bounds on the VC dimension and pseudodimension of feedforward neural networks composed of piecewise polynomial activation functions. We show that if the number of layers is fixed, then the VC dimension and pseudo-dimension grow as W log W, where W is the number of parameters in the network. This result stands in opposition to the case where the number of layers is unbounded, in which case the VC dimension and pseudo-dimension grow as W2. We combine our results with recently established approximation error rates and determine error bounds for the problem of regression estimation by piecewise polynomial networks with unbounded weights.

Approximation by neural networks and learning theory

We consider the problem of Learning Neural Networks from samples. The sample size which is sufficient for obtaining the almost-optimal stochastic approximation of function classes is obtained. In the terms of the accuracy confidence function, we show that the least-squares estimator is almost-optimal for the problem. These results can be used to solve Smales network problem.

Error bounds for functional approximation and estimation using mixtures of experts

We examine some mathematical aspects of learning unknown mappings with the mixture of experts model (MEM). Specifically, we observe that the MEM is at least as powerful as a class of neural networks, in a sense that will be made precise. Upper bounds on the approximation error are established for a wide class of target functions. The general theorem states that /spl par/f-f/sub n//spl par//sub p//spl les/c/n/sup r/d/ for f/spl isin/W/sub p//sup r/(L) (a Sobolev class over [-1,1]/sup d/), and f/sub n/ belongs to an n-dimensional manifold of normalized ridge functions. The same bound holds for the MEM as a special case of the above. The stochastic error, in the context of learning from independent and identically distributed (i.i.d.) examples, is also examined. An asymptotic analysis establishes the limiting behavior of this error, in terms of certain pseudo-information matrices. These results substantiate the intuition behind the MEM, and motivate applications.

On the near optimality of the stochastic approximation of smooth functions by neural networks

We consider the problem of approximating the Sobolev class of functions by neural networks with a single hidden layer, establishing both upper and lower bounds. The upper bound uses a probabilistic approach, based on the Radon and wavelet transforms, and yields similar rates to those derived recently under more restrictive conditions on the activation function. Moreover, the construction using the Radon and wavelet transforms seems very natural to the problem. Additionally, geometrical arguments are used to establish lower bounds for two types of commonly used activation functions. The results demonstrate the tightness of the bounds, up to a factor logarithmic in the number of nodes of the neural network.

Average n- widths of the Wiener space in the L ∞ -norm

Abstract Average n-widths of the Wiener space (C, w) in the L∞-norm have the asymptotic value dn(C, w, L∞) ≍ n− 1 2 .

The degree of approximation of sets in Euclidean space using sets with bounded Vapnik-Chervonenkis dimension

Abstract The degree of approximation of infinite-dimensional function classes using finite n-dimensional manifolds has been the subject of a classical field of study in the area of mathematical approximation theory. In Ratsaby and Maiorov (1997), a new quantity ρn(F, Lq) which measures the degree of approximation of a function class F by the best manifold Hn of pseudo-dimension less than or equal to n in the Lq-metric has been introduced. For sets F ⊂ R m it is defined as ρn(F, lmq) = infHn dist(F, Hn), where dist(F, Hn) = supxϵF infyϵHn∥x−y ∥lmq and H n ⊂ R m is any set of VC-dimension less than or equal to n where n H n ⊂ R m of VC-dimension less than or equal to n in the lmq-metric. In this paper we compute ρn(F, lmq) for F being the unit ball B m p = {x ϵ R m : ∥x∥ l m p ⩽ 1} for any 1 ⩽ p, q ⩽ ∞, and for F being any subset of the boolean m-cube of size larger than 2mγ, for any 1 2 .

On best approximation of classes by radial functions

We investigate the radial manifolds Rn, generated by a linear combination of n radial functions on Rd. We consider the best approximation of function classes by the manifold Rn. In particular, we prove that the deviation of the manifold Rn from the Sobolev class W2r,d in the Hilbert space L2 behaves asymptotically as n-r/d-1. We show the connection between the manifold Rn and the space of algebraic polynomials Pd,s of degree s. Namely, we prove there exist constants c1 and c2 such that the space Pd,s is either contained or not in Rn as n ≥ c1sd-1 or n ≤ c2sd-1, respectively.

On the Value of Partial Information for Learning from Examples

The PAC model of learning and its extension to real valued function classes provides a well-accepted theoretical framework for representing the problem of learning a target functiong(x) using a random sample {(xi,g(xi))}i=1m. Based on the uniform strong law of large numbers the PAC model establishes the sample complexity, i.e., the sample sizemwhich is sufficient for accurately estimating the target function to within high confidence. Often, in addition to a random sample, some form of prior knowledge is available about the target. It is intuitive that increasing the amount of information should have the same effect on the error as increasing the sample size. But quantitatively how does the rate of error with respect to increasing information compare to the rate of error with increasing sample size? To answer this we consider a new approach based on a combination of information-based complexity of Traubet al.and Vapnik?Chervonenkis (VC) theory. In contrast to VC-theory where function classes of finite pseudo-dimension are used only for statistical-based estimation, we let such classes play a dual role of functional estimation as well as approximation. This is captured in a newly introduced quantity, ?d(F), which represents a nonlinear width of a function class F. We then extend the notion of thenth minimal radius of information and define a quantityIn,d(F) which measures the minimal approximation error of the worst-case targetg? F by the family of function classes having pseudo-dimensiondgiven partial information ongconsisting of values taken bynlinear operators. The error rates are calculated which leads to a quantitative notion of the value of partial information for the paradigm of learning from examples.