Stronger connections between circuit analysis and circuit lower bounds, via PCPs of proximity

Abstract

We considerably sharpen the known connections between circuit-analysis algorithms and circuit lower bounds, show intriguing equivalences between the analysis of weak circuits and (apparently difficult) circuits, and provide strong new lower bounds for approximately computing Boolean functions with depth-two neural networks and related models. - We develop approaches to proving THR ○ THR lower bounds (a notorious open problem), by connecting algorithmic analysis of THR ○ THR to the provably weaker circuit classes THR ○ MAJ and MAJ ○ MAJ, where exponential lower bounds have long been known. More precisely, we show equivalences between algorithmic analysis of THR ○ THR and these weaker classes. The ε-error CAPP problem asks to approximate the acceptance probability of a given circuit to within additive error ε; it is the canonical derandomization problem. We show: - There is a non- trivial (2n/nω(1) time) 1/poly(n)-error CAPP algorithm for poly(n)-size THR ○ THR circuits if and only if there is such an algorithm for poly(n)-size MAJ ○ MAJ. - There is a δ > 0 and a non-trivial SAT (δ-error CAPP) algorithm for poly(n)-size THR ○ THR circuits if and only if there is such an algorithm for poly(n)-size THR ○ MAJ. Similar results hold for depth-d linear threshold circuits and depth-d MAJORITY circuits. These equivalences are proved via new simulations of THR circuits by circuits with MAJ gates. - We strengthen the connection between non-trivial derandomization (non-trivial CAPP algorithms) for a circuit class C, and circuit lower bounds against C. Previously, [Ben-Sasson and Viola, ICALP 2014] (following [Williams, STOC 2010]) showed that for any polynomial-size class C closed under projections, non-trivial (2n/nω(1) time) CAPP for ORpoly(n) ○ AND3 ○ C yields NEXP ⊄ C. We apply Probabilistic Checkable Proofs of Proximity in a new way to show it would suffice to have a non-trivial CAPP algorithm for either ⊕2 ○ C, AND2 ○ C or OR2 ○ C. - A direct corollary of the first two bullets is that NEXP ⊄ THR ○ THR would follow from either: - a non-trivial δ-error CAPP (or SAT) algorithm for poly(n)-size THR ○ MAJ circuits, or - a non-trivial 1 / poly(n)-error CAPP algorithm for poly(n)-size MAJ ○ MAJ circuits. - Applying the above machinery, we extend lower bounds for depth-two neural networks and related models [R. Williams, CCC 2018] to weak approximate computations of Boolean functions. For example, for arbitrarily small ε > 0, we prove there are Boolean functions f computable in nondeterministic nlog n time such that (for infinitely many n) every polynomial-size depth-two neural network N on n inputs (with sign or ReLU activation) must satisfy maxx∈{0,1}n |N(x) --- f(x)| > 1/2 --- ε. That is, short linear combinations of ReLU gates fail miserably at computing f to within close precision. Similar results are proved for linear combinations of ACC ○ THR circuits, and linear combinations of low-degree Fp polynomials. These results constitute further progress towards THR ○ THR lower bounds.