Dotan Di Castro

Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training

Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.

Adaptive bases for reinforcement learning

We consider the problem of reinforcement learning using function approximation, where the approximating basis can change dynamically while interacting with the environment. A motivation for such an approach is maximizing the value function fitness to the problem faced. Three errors are considered: approximation square error, Bellman residual, and projected Bellman residual. Algorithms under the actorcritic framework are presented, and shown to converge. The advantage of such an adaptive basis is demonstrated in simulations.

Structural Clustering of Machine-Generated Mail

Several recent studies have presented different approaches for clustering and classifying machine-generated mail based on email headers. We propose to expand these approaches by considering email message bodies. We argue that our approach can help increase coverage and precision in several tasks, and is especially critical for mail extraction. We remind that mail extraction supports a variety of mail mining applications such as ad re-targeting, mail search, and mail summarization. We introduce new structural clustering methods that leverage the HTML structure that is common to messages generated by a same mass-sender script. We discuss how such structural clustering can be conducted at different levels of granularity, using either strict or flexible matching constraints, depending on the use cases. We present large scale experiments carried over real Yahoo mail traffic. For our first use case of automatic mail extraction, we describe novel flexible-matching clustering methods that meet the key requirements of high intra-cluster similarity, adequate clusters size, and relatively small overall number of clusters. We identify the precise level of flexibility that is needed in order to achieve extremely high extraction precision (close to 100%), while producing relatively small number of clusters. For our second use case, namely, mail classification, we show that strict structural matching is more adequate, achieving precision and recall rates between 85%-90%, while converging to a stable classification after a short learning cycle. This represents an increase of 10%-20% compared to the sender-based method described in previous work, when run over the same period length. Our work has been deployed in production in Yahoo mail backend.

Adaptive bases for Q-learning

We consider reinforcement learning, and in particular, the Q-learning algorithm in large state and action spaces. In order to cope with the size of the spaces, a function approximation approach to the state and action value function is needed. We generalize the classical Q-learning algorithm to an algorithm where the basis of the linear function approximation change dynamically while interacting with the environment. A motivation for such an approach is maximizing the state-action value function fitness to the problem faced, thus obtaining better performance. The algorithm is shown to converge using two time scales stochastic approximation. Finally, we discuss how this technique can be applied to a rich family of RL algorithms with linear function approximation.

Automated Extractions for Machine Generated Mail.

Mail extraction is a critical task whose objective is to extract valuable data from the content of mail messages. This task is key for many types of applications including re-targeting, mail search, and mail summarization, which utilize the important personal data pieces in mail messages to achieve their objectives. We focus on machine generated traffic, which comprises most of the Web mail traffic today, and use its structured and large-scale repetitive nature to devise a fully automated extraction method. Our solution builds on an advanced structural clustering technique previously presented by some of the authors of this work. The heart of our solution is an offline process that leverages the structural mail-specific characteristics of the clustering, and automatically creates extraction rules that are later applied online for each new arriving message. We provide of a full description of our process, which has been productized in Yahoo mail backend. We complete our work with large-scale experiments carried over real Yahoo mail traffic, and evaluate the performance of our automatic extraction method.

Delays and oscillations in networks of spiking neurons: a two-timescale analysis

Oscillations are a ubiquitous feature of many neural systems, spanning many orders of magnitude in frequency. One of the most prominent oscillatory patterns, with possible functional implications, is that occurring in the mammalian thalamocortical system during sleep. This system is characterized by relatively long delays (reaching up to 40 msec) and gives rise to low-frequency oscillatory waves. Motivated by these phenomena, we study networks of excitatory and inhibitory integrate-and-fire neurons within a Fokker-Planck delay partial differential equation formalism and establish explicit conditions for the emergence of oscillatory solutions, and for the amplitude and period of the ensuing oscillations, for relatively large values of the delays. When a two-timescale analysis is employed, the full partial differential equation is replaced in this limit by a discrete time iterative map, leading to a relatively simple dynamic interpretation. This asymptotic result is shown numerically to hold, to a good approximation, over a wide range of parameter values, leading to an accurate characterization of the behavior in terms of the underlying physical parameters. Our results provide a simple mechanistic explanation for one type of slow oscillation based on delayed inhibition, which may play an important role in the slow spindle oscillations occurring during sleep. Moreover, they are consistent with experimental findings related to human motor behavior with visual feedback.

Correlated Rounding of Multiple Uniform Matroids and Multi-Label Classification

We introduce correlated randomized dependent rounding where, given multiple points y^1,...,y^n in some polytope P\subseteq [0,1]^k, the goal is to simultaneously round each y^i to some integral z^i in P while preserving both marginal values and expected distances between the points. In addition to being a natural question in its own right, the correlated randomized dependent rounding problem is motivated by multi-label classification applications that arise in machine learning, e.g., classification of web pages, semantic tagging of images, and functional genomics. The results of this work can be summarized as follows: (1) we present an algorithm for solving the correlated randomized dependent rounding problem in uniform matroids while losing only a factor of O(log{k}) in the distances (k is the size of the ground set); (2) we introduce a novel multi-label classification problem, the metric multi-labeling problem, which captures the above applications. We present a (true) O(log{k})-approximation for the general case of metric multi-labeling and a tight 2-approximation for the special case where there is no limit on the number of labels that can be assigned to an object.

Tutor learning using linear constraints in approximate dynamic programming

In adaptive control, agents interacting with Markov Decision Processes typically face two types of setups. In the first setup, the environments model is known and dynamic programming and related methods are used to obtain the optimal control. In the second setup, the environments model is unknown and reinforcement learning methods are used. In this work we investigate a new setup that is a mix of the two mentioned setups: only part of the environments model is known and additional information regarding the environment is provided by a tutor. We formalize this problem using linear function approximation in order to overcome the “curse of dimensionality” phenomenon. In addition, using the Envelope Theorem, we show how one can tune the approximation basis in order to get a locally optimal results. Finally, the suggested methods are demonstrated in simulations.