Tengyu Ma

Finding approximate local minima faster than gradient descent

We design a non-convex second-order optimization algorithm that is guaranteed to return an approximate local minimum in time which scales linearly in the underlying dimension and the number of training examples. The time complexity of our algorithm to find an approximate local minimum is even faster than that of gradient descent to find a critical point. Our algorithm applies to a general class of optimization problems including training a neural network and other non-convex objectives arising in machine learning.

Polynomial-Time Tensor Decompositions with Sum-of-Squares

We give new algorithms based on the sum-of-squares method for tensor decomposition. Our results improve the best known running times from quasi-polynomial to polynomial for several problems, including decomposing random overcomplete 3-tensors and learning overcomplete dictionaries with constant relative sparsity. We also give the first robust analysis for decomposing overcomplete 4-tensors in the smoothed analysis model. A key ingredient of our analysis is to establish small spectral gaps in moment matrices derived from solutions to sum-of-squares relaxations. To enable this analysis we augment sum-of-squaresrelaxations with spectral analogs of maximum entropy constraints.

Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms

Tensor rank and low-rank tensor decompositions have many applications in learning and complexity theory. Most known algorithms use unfoldings of tensors and can only handle rank up to

Generalization and equilibrium in generative adversarial nets (GANs) (invited talk)

n^{\lfloor p/2 \rfloor}

Provable Bounds for Learning Some Deep Representations

for a

Matrix Completion has No Spurious Local Minimum

p

A Simple but Tough-to-Beat Baseline for Sentence Embeddings

-th order tensor in

Generalization and Equilibrium in Generative Adversarial Nets (GANs)

\mathbb{R}^{n^p}

Identity Matters in Deep Learning

. Previously no efficient algorithm can decompose 3rd order tensors when the rank is super-linear in the dimension. Using ideas from sum-of-squares hierarchy, we give the first quasi-polynomial time algorithm that can decompose a random 3rd order tensor decomposition when the rank is as large as

Simple, efficient, and neural algorithms for sparse coding

n^{3/2}/\textrm{polylog} n