Yishay Mansour

A sparse sampling algorithm for near-optimal planning in large Markov decision processes

A critical issue for the application of Markov decision processes (MDPs) to realistic problems is how the complexity of planning scales with the size of the MDP. In stochastic environments with very large or infinite state spaces, traditional planning and reinforcement learning algorithms may be inapplicable, since their running time typically grows linearly with the state space size in the worst case. In this paper we present a new algorithm that, given only a generative model (a natural and common type of simulator) for an arbitrary MDP, performs on-line, near-optimal planning with a per-state running time that has no dependence on the number of states. The running time is exponential in the horizon time (which depends only on the discount factor γ and the desired degree of approximation to the optimal policy). Our algorithm thus provides a different complexity trade-off than classical algorithms such as value iteration—rather than scaling linearly in both horizon time and state space size, our running time trades an exponential dependence on the former in exchange for no dependence on the latter.Our algorithm is based on the idea of sparse sampling. We prove that a randomly sampled look-ahead tree that covers only a vanishing fraction of the full look-ahead tree nevertheless suffices to compute near-optimal actions from any state of an MDP. Practical implementations of the algorithm are discussed, and we draw ties to our related recent results on finding a near-best strategy from a given class of strategies in very large partially observable MDPs (Kearns, Mansour, & Ng. Neural information processing systems 13, to appear).

The shrinking generator

We present a new construction of a pseudorandom generator based on a simple combination of two LFSRs. The construction has attractive properties as simplicity (conceptual and implementation-wise), scalability (hardware and security), proven minimal security conditions (exponential period, exponential linear complexity, good statistical properties), and resistance to known attacks. The construction is suitable for practical implementation of efficient stream cipher cryptosystems.

An

We show that for any randomized broadcast protocol for radio networks, there exists a network in which the expected time to broadcast a message is

\Omega(D\log (N/D))

\Omega(D\log (N/D))

Lower Bound for Broadcast in Radio Networks

, where D is the diameter of the network and N is the number of nodes. This implies a tight lower bound of

On the learnability of discrete distributions

\Omega(D\log N)

Weakly learning DNF and characterizing statistical query learning using Fourier analysis

for any

A construction of a cipher from a single pseudorandom permutation

D \le N^{1-\varepsilon}

Learning Rates for Q-learning

, where

An experimental and theoretical comparison of model selection methods

\varepsilon > 0