David Rolnick

Why Does Deep and Cheap Learning Work So Well

We show how the success of deep learning could depend not only on mathematics but also on physics: although well-known mathematical theorems guarantee that neural networks can approximate arbitrary functions well, the class of functions of practical interest can frequently be approximated through “cheap learning” with exponentially fewer parameters than generic ones. We explore how properties frequently encountered in physics such as symmetry, locality, compositionality, and polynomial log-probability translate into exceptionally simple neural networks. We further argue that when the statistical process generating the data is of a certain hierarchical form prevalent in physics and machine learning, a deep neural network can be more efficient than a shallow one. We formalize these claims using information theory and discuss the relation to the renormalization group. We prove various “no-flattening theorems” showing when efficient linear deep networks cannot be accurately approximated by shallow ones without efficiency loss; for example, we show that n variables cannot be multiplied using fewer than

Quantitative Tverberg Theorems Over Lattices and Other Discrete Sets

2^n

On the growth of Stanley sequences

2n neurons in a single hidden layer.

Novel structures in Stanley sequences

An integer sequence is said to be 3-free if no three elements form an arithmetic progression. A Stanley sequence{an} is a 3-free sequence constructed by the greedy algorithm. Namely, given initial terms a0an1 is chosen to be the smallest such that the 3-free condition is not violated. Odlyzko and Stanley conjectured that Stanley sequences divide into two classes based on asymptotic growth: Type 1 sequences satisfy an=(nlog23) and appear well-structured, while Type 2 sequences satisfy an=(n2/logn) and appear disorderly. In this paper, we define the notion of regularity, which is based on local structure and implies Type 1 asymptotic growth. We conjecture that the reverse implication holds. We construct many classes of regular Stanley sequences, which include all known Type 1 sequences as special cases. We show how two regular sequences may be combined into another regular sequence, and how parts of a Stanley sequence may be translated while preserving regularity. Finally, we demonstrate the surprising fact that certain Stanley sequences possess proper subsets that are also Stanley sequences.

Quantitative Combinatorial Geometry for Continuous Parameters

This paper presents a new variation of Tverberg’s theorem. Given a discrete set S of

Quantitative Tverberg, Helly, & Carathéodory theorems

\mathbb {R}^{d}