Matthew J. Hausknecht

Beyond short snippets: Deep networks for video classification

Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%).

A Neuroevolution Approach to General Atari Game Playing

This paper addresses the challenge of learning to play many different video games with little domain-specific knowledge. Specifically, it introduces a neuroevolution approach to general Atari 2600 game playing. Four neuroevolution algorithms were paired with three different state representations and evaluated on a set of 61 Atari games. The neuroevolution agents represent different points along the spectrum of algorithmic sophistication - including weight evolution on topologically fixed neural networks (conventional neuroevolution), covariance matrix adaptation evolution strategy (CMA-ES), neuroevolution of augmenting topologies (NEAT), and indirect network encoding (HyperNEAT). State representations include an object representation of the game screen, the raw pixels of the game screen, and seeded noise (a comparative baseline). Results indicate that direct-encoding methods work best on compact state representations while indirect-encoding methods (i.e., HyperNEAT) allow scaling to higher dimensional representations (i.e., the raw game screen). Previous approaches based on temporal-difference (TD) learning had trouble dealing with the large state spaces and sparse reward gradients often found in Atari games. Neuroevolution ameliorates these problems and evolved policies achieve state-of-the-art results, even surpassing human high scores on three games. These results suggest that neuroevolution is a promising approach to general video game playing (GVGP).

For Want of a Nail: How Absences Cause Events.

Causation by omission is instantiated when an effect occurs from an absence, as in The absence of nicotine causes withdrawal or Not watering the plant caused it to wilt. The phenomenon has been viewed as an insurmountable problem for process theories of causation, which specify causation in terms of conserved quantities, like force, but not for theories that specify causation in terms of statistical or counterfactual dependencies. A new account of causation challenges these assumptions. According to the force theory, absences are causal when the removal of a force leads to an effect. Evidence in support of this account was found in 3 experiments in which people classified animations of complex causal chains involving force removal, as well as chains involving virtual forces, that is, forces that were anticipated but never realized. In a 4th experiment, the force theorys ability to predict synonymy relationships between different types of causal expressions provided further evidence for this theory over dependency theories. The findings show not only how causation by omission can be grounded in the physical world but also why only certain absences, among the potentially infinite number of absences, are causal.

HyperNEAT-GGP: a hyperNEAT-based atari general game player

This paper considers the challenge of enabling agents to learn with as little domain-specific knowledge as possible. The main contribution is HyperNEAT-GGP, a HyperNEAT-based General Game Playing approach to Atari games. By leveraging the geometric regularities present in the Atari game screen, HyperNEAT effectively evolves policies for playing two different Atari games, Asterix and Freeway. Results show that HyperNEAT-GGP outperforms existing benchmarks on these games. HyperNEAT-GGP represents a step towards the ambitious goal of creating an agent capable of learning and seamlessly transitioning between many different tasks.

Learning powerful kicks on the aibo ERS-7: the quest for a striker

Coordinating complex motion sequences remains a challenging task for robotics. Machine Learning has aided this process, successfully improving motion sequences such as walking and grasping. However, to the best of our knowledge, outside of simulation, learning has never been applied to the task of kicking the ball. We apply machine learning methods to optimize kick power entirely on a real robot. The resulting learned kick is significantly more powerful than the most powerful handcoded kick of one of the most successful RoboCup four-legged league teams, and is learned in a principled manner which requires very little engineering of the parameter space. Finally, model inversion is applied to the problem of creating a parameterized kick capable of kicking the ball a specified distance.

Dynamic lane reversal in traffic management

Contraflow lane reversal — the reversal of lanes in order to temporarily increase the capacity of congested roads — can effectively mitigate traffic congestion during rush hour and emergency evacuation. However, contraflow lane reversal deployed in several cities are designed for specific traffic patterns at specific hours, and do not adapt to fluctuations in actual traffic. Motivated by recent advances in autonomous vehicle technology, we propose a framework for dynamic lane reversal in which the lane directionality is updated quickly and automatically in response to instantaneous traffic conditions recorded by traffic sensors. We analyze the conditions under which dynamic lane reversal is effective and propose an integer linear programming formulation and a bi-level programming formulation to compute the optimal lane reversal configuration that maximizes the traffic flow. In our experiments, active contraflow increases network efficiency by 72%.

Using a million cell simulation of the cerebellum: Network scaling and task generality

Several factors combine to make it feasible to build computer simulations of the cerebellum and to test them in biologically realistic ways. These simulations can be used to help understand the computational contributions of various cerebellar components, including the relevance of the enormous number of neurons in the granule cell layer. In previous work we have used a simulation containing 12000 granule cells to develop new predictions and to account for various aspects of eyelid conditioning, a form of motor learning mediated by the cerebellum. Here we demonstrate the feasibility of scaling up this simulation to over one million granule cells using parallel graphics processing unit (GPU) technology. We observe that this increase in number of granule cells requires only twice the execution time of the smaller simulation on the GPU. We demonstrate that this simulation, like its smaller predecessor, can emulate certain basic features of conditioned eyelid responses, with a slight improvement in performance in one measure. We also use this simulation to examine the generality of the computation properties that we have derived from studying eyelid conditioning. We demonstrate that this scaled up simulation can learn a high level of performance in a classic machine learning task, the cart-pole balancing task. These results suggest that this parallel GPU technology can be used to build very large-scale simulations whose connectivity ratios match those of the real cerebellum and that these simulations can be used guide future studies on cerebellar mediated tasks and on machine learning problems.

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.

Detecting Stepping-Stone Intruders with Long Connection Chains

It is generally agreed that there is no valid reason to use a long connection chain for remote login such as SSH connection. Most of the stepping-stone detection algorithms installed on a host were designed to protect the victim of a third party downstream from where the algorithm is running. It is much more important for a host to protect itself from being a victim. This project uses an approximated round-trip time to distinguish a long connection chain from a short one. Several measures were studied to distinguish long chains from short ones. An estimated roundtrip time was defined to measure the chain length. Preliminary result suggests shows that the proposed algorithm can distinguish long connection chains from short ones with relatively low false rate.

Machine Learning Capabilities of a Simulated Cerebellum

This paper describes the learning and control capabilities of a biologically constrained bottom-up model of the mammalian cerebellum. Results are presented from six tasks: 1) eyelid conditioning; 2) pendulum balancing; 3) proportional–integral–derivative control; 4) robot balancing; 5) pattern recognition; and 6) MNIST handwritten digit recognition. These tasks span several paradigms of machine learning, including supervised learning, reinforcement learning, control, and pattern recognition. Results over these six domains indicate that the cerebellar simulation is capable of robustly identifying static input patterns even when randomized across the sensory apparatus. This capability allows the simulated cerebellum to perform several different supervised learning and control tasks. On the other hand, both reinforcement learning and temporal pattern recognition prove problematic due to the delayed nature of error signals and the simulator’s inability to solve the credit assignment problem. These results are consistent with previous findings which hypothesize that in the human brain, the basal ganglia is responsible for reinforcement learning, while the cerebellum handles supervised learning.