Tianshu Chu

Kernel-based reinforcement learning for traffic signal control with adaptive feature selection

Reinforcement learning in a large-scale system is computationally challenging due to the curse of the dimensionality. One approach is to approximate the Q-function as a function of a state-action related feature vector, then learn the parameters instead. Although assumptions from the priori knowledge can potentially explore an appropriate feature vector, selecting a biased one that insufficiently represents the system usually leads to the poor learning performance. To avoid this disadvantage, this paper introduces kernel methods to implicitly propose a learnable feature vector instead of a pre-selected one. More specifically, the feature vector is estimated from a reference set which contains all critical state-action pairs observed so far, and it can be updated by either adding a new pair or replace an existing one in the reference set. Thus the approximate Q-function keeps adjusting itself as the knowledge about the system accumulates via observations. Our algorithm is designed in both batch mode and online mode in the context of the traffic signal control. In addition, the convergence of this algorithm is experimentally supported. Furthermore, some regularization methods are proposed to avoid overfitting of Q-function on the noisy observations. Finally, A simulation on the traffic signal control in a single intersection is provided, and the performance of this algorithm is compared with Q-learning, in which the Q-function is numerically estimated for each state-action pair without approximation.

Traffic signal control with macroscopic fundamental diagrams

The recent breakthrough finding of macroscopic fundamental diagram (MFD) establishes the foundation of macroscopic analysis in urban transportation studies. However, the implementation of MFD for traffic signal control remains challenging. This is because the compact network-wide information provided by MFD is insufficient for searching for the optimal microscopic control policy. In this paper, rather than implementing only MFD, we integrate MFD into our microscopic urban traffic flow model to constrain the searching space of control policies. This approach is able to maximize the contribution of MFD, without losing microscopic information in the control model. Specifically, we first build a traffic flow model and introduce the stochastic driver behaviors by a turning matrix. We then implement the approximate Q-learning with restricted control to reduce the computational cost of the large-scale stochastic control problem. Here, the information of MFD is used to design both the heuristic regularization term in the stage cost and the statebased feature vector of the approximate Q-function. By this approximate Q-learning algorithm, the traffic density distribution of the network tends to become homogenous, with the mean value around the optimal density of the MFD. The numerical experiments demonstrate that compared to a fixed policy, our policy could efficiently make a heterogeneous network more homogeneous, and thus guarantee a more robust shape of the MFD. Furthermore, our policy has a better performance on trip completion flow maximization compared to either a fixed or a greedy policy, since it can achieve the optimal density in the MFD.

Large-scale traffic grid signal control with regional Reinforcement Learning

Reinforcement learning (RL) based traffic signal control for large-scale traffic grids is challenging due to the curse of dimensionality. Most particularly, searching for an optimal policy in a huge action space is impractical, even with approximate Q-functions. On the other hand, heuristic self-organizing algorithms could achieve efficient decentralized control, but most of them have few effort on optimizing the real-time traffic. This paper proposes a new regional RL algorithm that could form local cooperation regions adaptively, and then learn the optimal control policy for each region separately. In particular, we maintain a set of learning parameters to capture the control patterns in regions at different scales. At each time step, we first decompose the large-scale traffic grid into disjoint sub-regions, depending on the real-time traffic condition. Next, we apply approximate Q-learning to learn the centralized control policy within each sub-region, by updating the corresponding learning parameters upon traffic observations. The numerical experiments demonstrate that our regional RL algorithm is computationally efficient and functionally adaptive, and it outperforms typical heuristic decentralized algorithms.

Modeling and optimizing the performance of PVC/PVB ultrafiltration membranes using supervised learning approaches

Mathematical models play an important role in performance prediction and optimization of ultrafiltration (UF) membranes fabricated via dry/wet phase inversion in an efficient and economical manner. In this study, a systematic approach, namely, a supervised, learning-based experimental data analytics framework, is developed to model and optimize the flux and rejection rate of poly(vinyl chloride) (PVC) and polyvinyl butyral (PVB) blend UF membranes. Four supervised learning (SL) approaches, namely, the multiple additive regression tree (MART), the neural network (NN), linear regression (LR), and the support vector machine (SVM), are employed in a rigorous fashion. The dependent variables representing membrane performance response with regard to independent variables representing fabrication conditions are systematically analyzed. By comparing the predicting indicators of the four SL methods, the NN model is found to be superior to the other SL models with training and testing R-squared values as high as 0.8897 and 0.6344, respectively, for the rejection rate, and 0.9175 and 0.8093, respectively, for the flux. The optimal combination of processing parameters and the most favorable flux and rejection rate for PVC/PVB ultrafiltration membranes are further predicted by the NN model and verified by experiments. We hope the approach is able to shed light on how to systematically analyze multi-objective optimization issues for fabrication conditions to obtain the desired ultrafiltration membrane performance based on complex experimental data characteristics.

An Ontology-based Service Model for Smart Infrastructure Design

As a fundamental building block of smart cities, smart infrastructure has been increasingly drawing attention in both academia and industry across the globe. Many research efforts have been directed toward models for smart city infrastructure and city service. The existing models focus on either general description for features and criteria of smart infrastructure, or domain-specific investigation in an ad-hoc way, which, for example, applies optimization to smart transportation and smart buildings. However, a general model optimally augmenting smart features into traditional infrastructure is not reported in the literature so far. We propose such a service model for smart infrastructure designers to fill this gap in two steps. A static ontology of the service model is firstly setup to identify a general framework of the underlying infrastructure expressed by a multi-agent system (MAS) and a dynamic ontology is then designed to manage informed decision-making between global and local scales inside the system. With this approach we can accomplish the design of specific smart infrastructures based on expected goals, availability of information, and the feasibility of implementation. In order to demonstrate the effectiveness of the proposed approach we design and present two smart transportation systems as examples of this procedure. We also propose some future improvements of the service model.

An adaptive online learning framework for practical breast cancer diagnosis

This paper presents an adaptive online learning (OL) framework for supporting clinical breast cancer (BC) diagnosis. Unlike traditional data mining, which trains a particular model from a fixed set of medical data, our framework offers robust OL models that can be updated adaptively according to new data sequences and newly discovered features. As a result, our framework can naturally learn to perform BC diagnosis using experts’ opinions on sequential patient cases with cumulative clinical measurements. The framework integrates both supervised learning (SL) models for BC risk assessment and reinforcement learning (RL) models for decision-making of clinical measurements. In other words, online SL and RL interact with one another, and under a doctor’s supervision, push the patient’s diagnosis further. Furthermore, our framework can quickly update relevant model parameters based on current diagnosis information during the training process. Additionally, it can build flexible fitted models by integrating different model structures and plugging in the corresponding parameters during the prediction (or decision-making) process. Even when the feature space is extended, it can initialize the corresponding parameters and extend the existing model structure without loss of the cumulative knowledge. We evaluate the OL framework on real datasets from BCSC and WBC, and demonstrate that our SL models achieve accurate BC risk assessment from sequential data and incremental features. We also verify that the well-trained RL models provide promising measurement suggestions.

Traffic signal control by distributed Reinforcement Learning with min-sum communication

Reinforcement learning (RL) is a promising technique for adaptive signal control based on real-time traffic data. However, the implementation of RL in large-scale traffic signal network remains an open challenge due to the extremely high computational cost. In this paper, we integrate a distributed communication technique into RL so that the learning and searching cost of large-scale multi-agent system can be significantly reduced. In particular, we first decompose the global Q-function into local Q-functions, so each local signal agent can compute its own optimal policy based on local observations. We then apply the max-sum message passing algorithm to share information among agents, for finding a stable and optimal global policy that is accepted by all agents. Due to this information sharing characteristic, our approach is more optimal than decentralized RL, and it is much more efficient than centralized RL. Further, we perform realistic simulation to demonstrate that this distributed RL approach outperforms decentralized RL algorithm, as well as some typical heuristic decentralized control methods.

Large-Scale Traffic Grid Signal Control Using Decentralized Fuzzy Reinforcement Learning

With the rise of rapid urbanization around the world, a majority of countries have experienced a significant increase in traffic congestion. The negative impacts of this change have resulted in a number of serious and adverse effects, not only regarding the quality of daily life at an individual level but also for nations’ economic growth. Thus, the importance of traffic congestion management is well recognized. Adaptive real-time traffic signal control is effective for traffic congestion management. In particular, adaptive control with reinforcement learning (RL) is a promising technique that has recently been introduced in the field to better manage traffic congestion. Traditionally, most studies on traffic signal control have used centralized reinforcement learning, whose computation inefficiency prevents it from being employed for large traffic networks. In this paper, we propose a computationally cost-effective distributed algorithm, namely, a decentralized fuzzy reinforcement learning approach, to deal with problems related to the exponentially growing number of possible states and actions in RL models for a large-scale traffic signal control network. More specifically, the traffic density at each intersection is first mapped to four different fuzzy sets (i.e., low, medium, high, and extremely high). Next, two different kinds of algorithms, greedy and neighborhood approximate Q-learning (NAQL), are adaptively selected, based on the real-time, fuzzified congestion levels. To further reduce computational costs and the number of state-action pairs in the RL model, coordination and communication between the intersections are confined within a single neighborhood, i.e., the controlled intersection with its immediate neighbor intersections, for the NAQL algorithm. Finally, we conduct several numerical experiments to verify the efficiency and effectiveness of our approach. The results demonstrate that the decentralized fuzzy reinforcement learning algorithm achieves comparable results when measured against traditional heuristic-based algorithms. In addition, the decentralized fuzzy RL algorithm generates more adaptive control rules for the underlying dynamics of large-scale traffic networks. Thus, the proposed approach sheds new light on how to provide further improvements to a networked traffic signal control system for real-time traffic congestion.

A centralized reinforcement learning approach for proactive scheduling in manufacturing

Due to rapid development of information and communications technology (ICT) and the impetus for more effective, efficient and adaptive manufacturing, the concept of ICT based advanced manufacturing has increasingly become a prominent research topic across academia and industry during recent years. One critical aspect of advanced manufacturing is how to incorporate real time information and then optimally schedule manufacturing processes with multiple objectives. Due to its complexity and the need for adaptation, the manufacturing scheduling problem presents challenges for utilizing advanced ICT and thus calls for new approaches. The paper proposes a centralized reinforcement learning approach for optimally scheduling of a manufacturing system of multi-stage processes and multiple machines for multiple types of products. The approach, which employs learning and control algorithms to enable real time cooperation of each processing unit inside the system, is able to adaptively respond to dynamic scheduling changes. More specifically, we first formally define the scheduling problem through the construction of an objective function and related heuristic constraints for the underlying manufacturing tasks. Next, to effectively deal with the problem we defined, we maintain a distributed weighted vector to capture the cooperative pattern of massive action space and apply the reinforcement-learning approach to achieve the optimal policies for a set of processing machines according to a real time production environment, including dynamic requests for various products. Numerical experiments demonstrate that compared to different heuristic methods and multi-agent algorithms, the proposed centralized reinforcement learning method can provide more reliable solutions for the scheduling problem.

Facilitating Multiple-Objective Decision-Making for Advanced Manufacturing: A Knowledge Representation and Computational Active Learning-Based Simulation Framework

Due to the rapid development of information technology and the impetus for more efficient and adaptive manufacturing processes, the concept of advanced manufacturing has become an increasingly prominent research topic across academia and industry in recent years. One critical aspect of advanced manufacturing is how to optimally cope with the complexities of multiple-objective decision-making to implement advanced manufacturing technologies with currently available enterprise resources and the realistic manufacturing conditions of a company. Generally, to successfully fulfill an advanced manufacturing plan, decision-makers must align short-term objectives with long-term strategies. In addition, the decision-making process usually has to prioritize multiple-objective goals under a considerable number of uncertainties. This requirement presents new challenges for both planning and implementing advanced manufacturing technologies, and thus calls for new approaches for to better support such tasks. This paper proposes a knowledge representation and computational active learning-based framework for dealing with complex, multiple-objective decision-making problems for advanced manufacturing under realistic conditions. Through this study, we hope to shed light on using a simulation framework for multiple-objective decision support, thereby providing an alternative for manufacturing enterprises, which could lead to an acceptable optimal decision with reasonable cost and accuracy. First, we describe the scope of an advanced manufacturing system for industrial manufacturing. Next, we introduce systematic analysis of the complexities of the decision-making to implement advanced manufacturing. Finally, we propose a simulation model for the decision-making and formulate a computational active learning-based framework to efficiently compute goal priorities for multiple-objective decision-making. We validate the framework by presenting a simulation of decision-making.Copyright