Steve Smale

On the mathematical foundations of learning

(1) A main theme of this report is the relationship of approximation to learning and the primary role of sampling (inductive inference). We try to emphasize relations of the theory of learning to the mainstream of mathematics. In particular, there are large roles for probability theory, for algorithms such as least squares, and for tools and ideas from linear algebra and linear analysis. An advantage of doing this is that communication is facilitated and the power of core mathematics is more easily brought to bear. We illustrate what we mean by learning theory by giving some instances. (a) The understanding of language acquisition by children or the emergence of languages in early human cultures. (b) In Manufacturing Engineering, the design of a new wave of machines is anticipated which uses sensors to sample properties of objects before, during, and after treatment. The information gathered from these samples is to be analyzed by the machine to decide how to better deal with new input objects (see [43]). (c) Pattern recognition of objects ranging from handwritten letters of the alphabet to pictures of animals, to the human voice. Understanding the laws of learning plays a large role in disciplines such as (Cognitive) Psychology, Animal Behavior, Economic Decision Making, all branches of Engineering, Computer Science, and especially the study of human thought processes (how the brain works). Mathematics has already played a big role towards the goal of giving a universal foundation of studies in these disciplines. We mention as examples the theory of Neural Networks going back to McCulloch and Pitts [25] and Minsky and Papert [27], the PAC learning of Valiant [40], Statistical Learning Theory as developed by Vapnik [42], and the use of reproducing kernels as in [17] among many other mathematical developments. We are heavily indebted to these developments. Recent discussions with a number of mathematicians have also been helpful. In

Emergent Behavior in Flocks

We provide a model (for both continuous and discrete time) describing the evolution of a flock. Our model is parameterized by a constant beta capturing the rate of decay-which in our model is polynomial-of the influence between birds in the flock as they separate in space. Our main result shows that when beta<1/2 convergence of the flock to a common velocity is guaranteed, while for betages1/2 convergence is guaranteed under some condition on the initial positions and velocities of the birds only

Mathematical problems for the next century

V. I. Arnold, on behalf of the International Mathematical Union has written to a number of mathematicians with a suggestion that they describe some great problems for the next century. This report is my response. Arnolds invitation is inspired in part by Hilberts list of 1900 (see e.g. [Browder, 1976]) and I have used that list to help design this essay. I have listed 18 problems, chosen with these criteria:

The Mathematics of Learning: Dealing With Data

Abstract Learning is key to developing systems tailored to a broad range of data analysis and information extraction tasks. We outline the mathematical foundations of learning theory and describe a key algorithm of it.

A convergent process of price adjustment and global newton methods

Section 1 One goal of this paper is to give a relation between such diverse parts of economic theory as the Arrow-Block-Hurwicz dynamics of price adjustment and Scarf’s algorithm for finding economic equilibria. The underlying concept is an ordinary differential equation, which we call a ‘Global Newton’ one, associated to a system of IZ real functions,f, , . . . ,f,, of n real variables, x1, . . . , x,. The key feature of this differential equation is that, under suitable hypotheses, its solution will tend to a vector (XT, . . . , AT,*) satisfying fi(

On the Structure of Manifolds

, . . ., x,*) = 0, . . .,h(xT, . . .) x,*) = 0, (1) or in vector notation, f(x*) = 0. (1’) In fact in this way an algorithm for solving (1) is provided. The differential equation itself has the form

Newton’s Method Estimates from Data at One Point

In this paper, we prove a number of theorems which give some insight into the structure of differentiable manifolds. The methods, results and some notation of [13], hereafter referred to as GPC, and [12] will be used. These two papers and [14] can be considered as a starting point for this one. The main theorems in these papers are special cases of the theorems here. Among the most important theorems in this paper are 1. 1 and 6. 1. Some conversations with A. Haefliger were helpful in the preparation of parts of this paper. Everything will be considered from the differentable, equivalently C*, point of view; manifolds, imbeddings, and isotopes will be C*.

On the efficiency of algorithms of analysis

Newton’s method and its modifications have long played a central role in finding solutions of non-linear equations and systems. The work of Kantorovich has been seminal in extending and codifying Newton’s method. Kantorovich’s approach, which dominates the literature in this area, has these features: (a) weak differentiability hypotheses are made on the system, e.g., the map is C 2 on some domain in a Banach space; (b) derivative bounds are supposed to exist over the whole of this domain. In contrast, here strong hypotheses on differentiability are made; analyticity is assumed. On the other hand, we deduce consequences from data at a single point. This point of view has valuable features for computation and its theory. Theorems similar to ours could probably be deduced with the Kantorovich theory as a starting point; however, we have found it useful to start afresh.

On the differential equations of species in competition

CHAPTER I

Shannon sampling and function reconstruction from point values

SummaryIt is shown that the ordinary differential equation commonly used to describe competing species are compatible with any dynamical behavior provided the number of species in very large.