J. Zico Kolter

Towards fully autonomous driving: Systems and algorithms

In order to achieve autonomous operation of a vehicle in urban situations with unpredictable traffic, several realtime systems must interoperate, including environment perception, localization, planning, and control. In addition, a robust vehicle platform with appropriate sensors, computational hardware, networking, and software infrastructure is essential.

Near-Bayesian exploration in polynomial time

We consider the exploration/exploitation problem in reinforcement learning (RL). The Bayesian approach to model-based RL offers an elegant solution to this problem, by considering a distribution over possible models and acting to maximize expected reward; unfortunately, the Bayesian solution is intractable for all but very restricted cases. In this paper we present a simple algorithm, and prove that with high probability it is able to perform ε-close to the true (intractable) optimal Bayesian policy after some small (polynomial in quantities describing the system) number of time steps. The algorithm and analysis are motivated by the so-called PAC-MDP approach, and extend such results into the setting of Bayesian RL. In this setting, we show that we can achieve lower sample complexity bounds than existing algorithms, while using an exploration strategy that is much greedier than the (extremely cautious) exploration of PAC-MDP algorithms.

Regularization and feature selection in least-squares temporal difference learning

We consider the task of reinforcement learning with linear value function approximation. Temporal difference algorithms, and in particular the Least-Squares Temporal Difference (LSTD) algorithm, provide a method for learning the parameters of the value function, but when the number of features is large this algorithm can over-fit to the data and is computationally expensive. In this paper, we propose a regularization framework for the LSTD algorithm that overcomes these difficulties. In particular, we focus on the case of l1 regularization, which is robust to irrelevant features and also serves as a method for feature selection. Although the l1 regularized LSTD solution cannot be expressed as a convex optimization problem, we present an algorithm similar to the Least Angle Regression (LARS) algorithm that can efficiently compute the optimal solution. Finally, we demonstrate the performance of the algorithm experimentally.

The Stanford LittleDog: A learning and rapid replanning approach to quadruped locomotion

Legged robots have the potential to navigate a wide variety of terrain that is inaccessible to wheeled vehicles. In this paper we consider the planning and control tasks of navigating a quadruped robot over challenging terrain, including terrain that it has not seen until run-time. We present a software architecture that makes use of both static and dynamic gaits, as well as specialized dynamic maneuvers, to accomplish this task. Throughout the paper we highlight two themes that have been central to our approach: (1) the prevalent use of learning algorithms, and (2) a focus on rapid recovery and replanning techniques; we present several novel methods and algorithms that we developed for the quadruped and that illustrate these two themes. We evaluate the performance of these different methods, and also present and discuss the performance of our system on the official Learning Locomotion tests.

Stereo vision and terrain modeling for quadruped robots

Legged robots offer the potential to navigate highly challenging terrain, and there has recently been much progress in this area. However, a great deal of this recent work has operated under the assumption that either the robot has complete knowledge of its environment or that its environment is suitably regular so as to be navigated with only minimal perception, an unrealistic assumption in many real-world domains. In this paper we present an integrated perception and control system for a quadruped robot that allows it to perceive and traverse previously unseen, rugged terrain that includes large, irregular obstacles. A key element of the system is a novel terrain modeling algorithm, used for filling in the occluded models resulting from on-board vision systems. We apply our approach to the LittleDog robot, and show that it allows the robot to walk over challenging terrain using only on-board perception.

A probabilistic approach to mixed open-loop and closed-loop control, with application to extreme autonomous driving

We consider the task of accurately controlling a complex system, such as autonomously sliding a car sideways into a parking spot. Although certain regions of this domain are extremely hard to model (i.e., the dynamics of the car while skidding), we observe that in practice such systems are often remarkably deterministic over short periods of time, even in difficult-to-model regions. Motivated by this intuition, we develop a probabilistic method for combining closed-loop control in the well-modeled regions and open-loop control in the difficult-to-model regions. In particular, we show that by combining 1) an inaccurate model of the system and 2) a demonstration of the desired behavior, our approach can accurately and robustly control highly challenging systems, without the need to explicitly model the dynamics in the most complex regions and without the need to hand-tune the switching control law. We apply our approach to the task of autonomous sideways sliding into a parking spot, and show that we can repeatedly and accurately control the system, placing the car within about 2 feet of the desired location; to the best of our knowledge, this represents the state of the art in terms of accurately controlling a vehicle in such a maneuver.

Task-space trajectories via cubic spline optimization

We consider the task of planning smooth trajectories for robot motion. In this paper we make two contributions. First we present a method for cubic spline optimization; this technique lets us simultaneously plan optimal task-space trajectories and fit cubic splines to the trajectories, while obeying many of the same constraints imposed by a typical motion planning algorithm. The method uses convex optimization techniques, and is therefore very fast and suitable for real-time re-planning and control. Second, we apply this approach to the tasks of planning foot and body trajectory for a quadruped robot, the “LittleDog,” and show that the proposed approach improves over previous work on this robot.

Large-scale probabilistic forecasting in energy systems using sparse Gaussian conditional random fields

Short-term forecasting is a ubiquitous practice in a wide range of energy systems, including forecasting demand, renewable generation, and electricity pricing. Although it is known that probabilistic forecasts (which give a distribution over possible future outcomes) can improve planning and control, many forecasting systems in practice are just used as “point forecast” tools, as it is challenging to represent high-dimensional non-Gaussian distributions over multiple spatial and temporal points. In this paper, we apply a recently-proposed algorithm for modeling high-dimensional conditional Gaussian distributions to forecasting wind power and extend it to the non-Gaussian case using the copula transform. On a wind power forecasting task, we show that this probabilistic model greatly outperforms other methods on the task of accurately modeling potential distributions of power (as would be necessary in a stochastic dispatch problem, for example).

A moving horizon state estimator in the control of thermostatically controlled loads for demand response

The quality and effectiveness of the load following services provided by centralized control of thermostatically controlled loads depend highly on the communication requirements and the underlying cyberinfrastructure characteristics. Specifically, ensuring end-user comfort while providing real-time demand response services depends on the availability of the information provided from the thermostatically controlled loads to the main controller regarding their operating statuses and internal temperatures. State estimation techniques can be used to infer the necessary information from the aggregate power consumption of these loads, replacing the need for an upstream communication platform carrying information from appliances to the main controller in real-time. In this paper, we introduce a moving horizon mean squared error state estimator with constraints as an alternative to a Kalman filter approach, which assumes a linear model without constraints. The results show that some improvement is possible for scenarios when loads are expected to be toggled frequently.

Optimal Planning and Learning in Uncertain Environments for the Management of Wind Farms

AbstractWind energy is a key renewable source, yet wind farms have relatively high cost compared with many traditional energy sources. Among the life cycle costs of wind farms, operation and maintenance (O&M) accounts for 25–30%, and an efficient strategy for management of turbines can significantly reduce the O&M cost. Wind turbines are subject to fatigue-induced degradation and need periodic inspections and repairs, which are usually performed through semiannual scheduled maintenance. However, better maintenance can be achieved by flexible policies based on prior knowledge of the degradation process and on data collected in the field by sensors and visual inspections. Traditional methods to model the O&M process, such as Markov decision processes (MDPs) and partially observable MDPs (POMDPs), have limitations that do not allow the model to properly include the knowledge available and that may result in nonoptimal strategies for management of the farm. Specifically, the conditional probabilities for mode...