Traffic Signal Control With Reinforcement Learning Based on Region-Aware Cooperative Strategy

Abstract

With the increase of private cars, traditional traffic signal control methods cannot alleviate the traffic congestion problem. Reinforcement learning (RL) is increasingly used in adaptive traffic light control. As urban traffic becomes more complex, reinforcement learning algorithms solely based on value or policy are not suitable for such scenarios. Moreover, the centralized method does not show a good effect on the multi-intersection, in particular, when the traffic flow is in the high-dimensional continuous state space. In this article, we propose a decentralized framework based on the advantage actor-critic (A2C) algorithm by assigning global control to each local RL agent or intersection. A2C algorithm involved in this article is a product of combining policy function and value function, which has good convergence ability and can be applied to continuous state space. The decentralized methods may put a new challenge: from the perspective of each local agent, the environment becomes partially observable. We overcome this problem by putting forward a region-aware cooperative strategy (RACS) based on graph attention network (GAT), which can incorporate the spatial information of the surrounding agents. We carry out experiments on synthetic traffic grid and real-world traffic network of Monaco city to compare with the existing A2C and Q-learning algorithms. Experimental results confirm that our RACS method has a shorter queue length and less waiting time than the two existing algorithms, and can reduce the total vehicle travel time.