Samah El-Tantawy

Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto

Population is steadily increasing worldwide, resulting in intractable traffic congestion in dense urban areas. Adaptive traffic signal control (ATSC) has shown strong potential to effectively alleviate urban traffic congestion by adjusting signal timing plans in real time in response to traffic fluctuations to achieve desirable objectives (e.g., minimize delay). Efficient and robust ATSC can be designed using a multiagent reinforcement learning (MARL) approach in which each controller (agent) is responsible for the control of traffic lights around a single traffic junction. Applying MARL approaches to the ATSC problem is associated with a few challenges as agents typically react to changes in the environment at the individual level, but the overall behavior of all agents may not be optimal. This paper presents the development and evaluation of a novel system of multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC). MARLIN-ATSC offers two possible modes: 1) independent mode, where each intersection controller works independently of other agents; and 2) integrated mode, where each controller coordinates signal control actions with neighboring intersections. MARLIN-ATSC is tested on a large-scale simulated network of 59 intersections in the lower downtown core of the City of Toronto, ON, Canada, for the morning rush hour. The results show unprecedented reduction in the average intersection delay ranging from 27% in mode 1 to 39% in mode 2 at the network level and travel-time savings of 15% in mode 1 and 26% in mode 2, along the busiest routes in Downtown Toronto.

An agent-based learning towards decentralized and coordinated traffic signal control

Adaptive traffic signal control is a promising technique for alleviating traffic congestion. Reinforcement Learning (RL) has the potential to tackle the optimal traffic control problem for a single agent. However, the ultimate goal is to develop integrated traffic control for multiple intersections. Integrated traffic control can be efficiently achieved using decentralized controllers. Multi-Agent Reinforcement Learning (MARL) is an extension of RL techniques that makes it possible to decentralize multiple agents in a non-stationary environments. Most of the studies in the field of traffic signal control consider a stationary environment, an approach whose shortcomings are highlighted in this paper. A Q-Learning-based acyclic signal control system that uses a variable phasing sequence is developed. To investigate the appropriate state model for different traffic conditions, three models were developed, each with different state representation. The models were tested on a typical multiphase intersection to minimize the vehicle delay and were compared to the pre-timed control strategy as a benchmark. The Q-Learning control system consistently outperformed the widely used Webster pre-timed optimized signal control strategy under various traffic conditions.

Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC)

Traffic congestion in Greater Toronto Area costs Canada

Safety Evaluation of Truck Lane Restriction Strategies Using Microsimulation Modeling

6 billion /year and is expected to grow up to

Design of Reinforcement Learning Parameters for Seamless Application of Adaptive Traffic Signal Control

15 billion /year in the next few decades. Adaptive Traffic Signal Control(ATSC) is a promising technique to alleviate traffic congestion. For medium-large transportation networks, coordinated ATSC is becoming a challenging problem because the number of system states and actions grows exponentially as the number of networked intersections grows. Efficient and robust controllers can be designed using a multi-agent reinforcement learning (MARL) approach in which each controller (agent) is responsible for the control of traffic lights around a single traffic junction. This paper presents a novel, decentralized and coordinated adaptive real-time traffic signal control system using Multi-Agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLINATSC) that aims to minimize the total vehicle delay in the traffic network. The system is tested using microscopic traffic simulation software (PARAMICS) on a network of 5 signalized intersections in Downtown Toronto. The performance of MARLIN-ATSC is compared against two approaches: the conventional pretimed signal control (B1) and independent RL-based control agents (B2), i.e. with no coordination. The results show that network-wide average delay savings range from 32% to 63% relative to B1 and from 7% to 12% relative to B2 under different demand levels and arrival profiles.

Frame-based mobility estimation via compressive sensing in delay-tolerant vehicular networks

The demand for goods and services in North America has increased dramatically over the past few decades. This demand increase resulted in an associated increase in truck traffic on North American highways, escalating congestion, operation, and safety concerns. This in turn has led to increasing interest in strategies to reduce interaction between trucks and cars, including truck restrictions and dedicated truck lanes. The purpose of this paper is to implement algorithms for evaluating safety measures for different scenarios of truck lane restrictions and dedicated truck lanes using microscopic traffic simulation. The measures of performance calculated in this study are lane changing, merging, and rear-end conflicts. These measures are analyzed for truck-restricted lanes and dedicated truck lanes on the Gardiner Expressway in downtown Toronto, Canada. Simulation scenarios are developed by varying lane strategies and truck percentage. Simulation results showed that implementation of a single truck-restricted lane makes little difference to most conflict measures. Restricting trucks from the leftmost two lanes results in more substantial reductions in lane changing conflicts, but causes some increased freeway merging conflicts involving trucks. Dedicating the leftmost lane only to trucks also reduces lane changing conflicts but increases lane merging conflicts. Because lane changing conflicts are far more frequent than merging conflicts, there appears to be a net safety benefit by either restricting trucks from the left two lanes or dedicating the left lane to trucks. Relationships between lane strategy and rear-end conflicts are weak. Truck lane strategies are most effective when truck percentage exceeds 15%.

Assessment of Adaptive Traffic Signal Control Using Hardware in the Loop Simulation

Adaptive traffic signal control (ATSC) is a promising technique to alleviate traffic congestion. This article focuses on the development of an adaptive traffic signal control system using Reinforcement Learning (RL) as one of the efficient approaches to solve such stochastic closed loop optimal control problem. A generic RL control engine is developed and applied to a multi-phase traffic signal at an isolated intersection in Downtown Toronto in a simulation environment. Paramics, a microscopic simulation platform, is used to train and evaluate the adaptive traffic control system. This article investigates the following dimensions of the control problem: 1) RL learning methods, 2) traffic state representations, 3) action selection methods, 4) traffic signal phasing schemes, 5) reward definitions, and 6) variability of flow arrivals to the intersection. The system was tested on three networks (i.e., small, medium, large-scale) to ensure seamless transferability of the system design and results. The RL controller is benchmarked against optimized pretimed control and actuated control. The RL-based controller saves 48% average vehicle delay when compared to optimized pretimed controller and fully-actuated controller. In addition, the effect of the best design of RL-based ATSC system is tested on a large-scale application of 59 intersections in downtown Toronto and the results are compared versus the base case scenario of signal control systems in the field which are mix of pretimed and actuated controllers. The RL-based ATSC results in the following savings: average delay (27%), queue length (28%), and l CO2 emission factors (28%).

Traffic Data Fusion Using SCAAT Kalman Filters

In this paper, we study the problem of estimating mobility trajectories via a small number of measurements from realistic mobility traces. We propose a frame-based solution to estimate the complete mobility trace at the end of the frame. During each frame, a few measurements are transmitted via vehicular communications, and then estimation is performed at the receiver. We propose different sampling schemes to estimate the mobility trajectory at the end of the frame, and we compare it to the actual mobility trace. The proposed scheme is designed as a practical system for vehicular mobility estimation, and is suitable for road traffic management applications. We study the proposed mobility estimation scheme via simulations with realistic mobility traces generated by Paramics simulator [1]. The data are based on a realistic map of the Gardinar Expressway in the city of Toronto. Extensive simulation results shows that the proposed scheme can significantly reduce the number of transmitted samples while providing a good estimate of the mobility traces.

Towards multi-agent reinforcement learning for integrated network of optimal traffic controllers (MARLIN-OTC)

Adaptive Traffic Signal Control (ATSC) can potentially mitigate traffic congestion. Research and development in the area of ATSC have produced a number of new systems with promising potential, however the performance of these systems under real-life conditions has been always a concern for practitioners as well as researchers, particularly if and how the new systems would be implementable in the field on controllers with specific capabilities and limitations. Therefore, testing and refining new ATSC systems on actual hardware and under representative traffic conditions prior to field implementation is essential to bridge this gap. In this paper, a hardware in-the-loop simulation (HILS) framework is developed to evaluate MARLIN, as an example of a new self-learning ATSC system. HILS is used for evaluating hardware components running the ATSC software in a simulation environment in which an actual traffic signal controller and an embedded computer are physically connected to a microscopic traffic simulator. Our focus is on the development, implementation of the HILS framework and the evaluation of MARLIN, on an intersection that suffers significant traffic fluctuation and delays - at the City of Burlington, Ontario, Canada. The performance of MARLIN-ATSC is demonstrated with HILS, which consists of a PEEK ATC-1000 traffic controller, an embedded computer running the ATSC system, and Paramics microscopic simulation model. HILS results indicated that MARLIN-ATSC has the potential to reduce the intersection average delay by up to 20% on average compared to the optimized and coordinated actuated signal timing plans.