Spatio-Temporal Look-Ahead Trajectory Prediction using Memory Neural Network
SSpatio-Temporal Look-Ahead Trajectory Predictionusing Memory Neural Network
Nishanth Rao
Dept. of Aerospace EngineeringIndian Institute of Science
Bangalore, [email protected]
Suresh Sundaram
Dept. of Aerospace EngineeringIndian Institute of Science
Bangalore, [email protected]
Abstract —Prognostication of vehicle trajectories in unknownenvironments is intrinsically a challenging and difficult problemto solve. The behavior of such vehicles is highly influenced by sur-rounding traffic, road conditions, and rogue participants presentin the environment. Moreover, the presence of pedestrians, trafficlights, stop signs, etc., makes it much harder to infer the behaviorof various traffic agents. This paper attempts to solve the problemof spatio (cid:57) temporal look (cid:57) ahead trajectory prediction using a novelrecurrent neural network called the Memory Neuron Network.The Memory Neuron Network (MNN) attempts to capture theinput-output relationship between the past positions and thefuture positions of the traffic agents. The proposed model iscomputationally less intensive and has a simple architecture ascompared to other deep learning models that utilize LSTMs andGRUs. It is then evaluated on the publicly available NGSIMdataset and its performance is compared with several state (cid:57) of (cid:57) artalgorithms. Additionally, the performance is also evaluated on acustom synthetic dataset generated from the CARLA simulator.It is seen that the proposed model outperforms the existingstate (cid:57) of (cid:57) art algorithms. Finally, the model is integrated withthe CARLA simulator to test its robustness in real (cid:57) time trafficscenarios.
I. I
NTRODUCTION
Research in autonomous vehicles has attracted a lot ofinterest from researchers around the world. With the riseof electric vehicles over the past few years, autonomousnavigation and path planning have become an inherent featureof these vehicles. In presence of traffic, these vehicles shouldreach their destination and also follow traffic rules, preventaccidents, detect various traffic signs, handle reckless driversand rogue vehicles. To be able to perform the aforementionedtasks, the autonomous vehicle must have the ability to predictthe motion of it’s surrounding vehicles. This will enablethe vehicle to make necessary decisions at the right time.Anticipating traffic scenarios is thus a major functionality ofautonomous vehicles in order to navigate safely amidst theirhuman counterparts.This is a very challenging problem due to the unpredictablenature of traffic agents. Their behaviour is often determined bymultiple latent variables that cannot be estimated beforehandin new and unknown environments, such as the mental stateand driving experiences of human drivers, road and weatherconditions, destination of each vehicle in the traffic, reckless behaviour of traffic agents that involve overtaking, abrupt lanechanging without indication, etc.Many recent state (cid:57) of (cid:57) art deep learning models have utilized Long-Short Term Memory (LSTM) networks [1] and
GatedRecurrent Units (GRUs) [2] for the trajectory prediction prob-lem. One technique that is utilized by many approaches isthat of an encoder-decoder architecture. In these approaches,the spatio-temporal context from the vehicle trajectories isextracted and then a recurrent neural network (RNN) baseddecoder is used to predict the future trajectories. While theyhave been successful in regressing the future trajectories oftraffic agents over a certain time horizon, they are heavilydependent on computational resources due to their complexarchitecture and require a lot of training time.This paper will attempt to address all the aforementionedproblems by adapting a unique recurrent neural network calledthe Memory Neuron Network [3]. The Memory Neuron Net-work is an extension of the traditional neural network withaddition of memory elements to each neuron in the network,that are capable of storing temporal information. This networkhas a simple architecture, and requires less computationalresources as compared to the currently available state (cid:57) of (cid:57) artdeep learning methods. The performance of the proposedmodel is evaluated on the publicly available NGSIM US-101dataset. Although, NGSIM dataset provide comprehensive dataof real traffic agents, it does not contain sufficient data forreckless and rogue traffic agents. To address this situation, asynthetic dataset is generated using the CARLA simulator [4]that contains the trajectories of multiple heterogeneous roguetraffic agents. As the proposed model is computationally lessintensive, it allows for the deployment onto all the rogue vehi-cles present in the real (cid:57) time traffic simulation with additional80 normal cars. To summarize, our main contributions are asfollows: • A novel model is proposed that uses a recurrent neuralnetwork (cid:57) the Memory Neuron Network for the problemof spatio (cid:57) temporal look (cid:57) ahead trajectory prediction. • The proposed model is evaluated on publicly availableUS (cid:57)
101 dataset, and the RMSE is reported along withseveral state (cid:57) of (cid:57) art methods. • To evaluate the performance of our model with respect toreckless drivers, rogue vehicles are simulated on
CARLA a r X i v : . [ c s . R O ] F e b imulator and their trajectories are recorded. The model isthen implemented in real (cid:57) time simulation on each roguevehicle with a look (cid:57) ahead horizon of s , demonstratingthe robustness and the computational efficiency of theproposed model.II. R ELATED W ORK
This section sets out to explore some of the various methodscurrently present in the literature to address the motion predic-tion problem. The existing literature can be broadly classifiedinto three parts which are discussed below.
A. Mechanics-based methods
In these approaches, vehicles are mathematically modelledusing Newtonian laws of translation and rotation. Once themodel is formed, an Unscented Kalman Filter (UKF) is usedto estimate the states of the vehicles. [5] propose an InteractiveMultiple Model Trajectory Prediction (IMMTP) which com-bines physics-based and manoeuvre-based predictive models.[6] use a deterministic sampling approach in the UKF processfor a robust estimate of target trajectories. These models workreally well in certain scenarios and short time predictionhorizon. However, these approaches tend to linearize theobtained models and hence, are unable to capture the inherentnon-linear characteristics in a generic traffic scenario. Anotherissue with these approaches is that the parameters of themathematical model such as the dimensions of the vehicle, itsbraking coefficients, steering torque etc., must be set and tunedin real (cid:57) time, as soon as a vehicle is detected in the vicinity.This may not be feasible when the other agent’s model isunknown. A detailed study on these methods can be found in[7].
B. Human behavior-based models
These techniques attempt to build a mathematical formu-lation of the human behavior and utilize these as a modelfor the driving process. [8] apply the theory of plannedbehavior to model the driver behavior, and develop a drivermodel that accounts for various human aspects such as drivingexperiences, emotions, age, gender etc. [9] and [10] applycontrol theory and Markov Decision Process (MDP) to modelhuman behaviors specifically for the navigation process in asingle lane. To extend the analysis to multi (cid:57) lane junctions,Hidden Markov Models are proposed to model human behav-iors in [11]. Statistical models have been proposed in [12],[13] and [14] to predict driving manoeuvres and behaviors.These methods work best when the knowledge of the humanbehaviors and their analysis are known beforehand. However,in the case of new and unknown environments these modelsfail to provide reliable predictions.
C. Deep learning methods
These methods use a spatial encoder to process the raw tra-jectory data, and then use recurrent neural networks to estimatethe future trajectories. To extract the spatial context from thetrajectories, [15], [16] and [17] use a sequential point (cid:57) based TrajectoryDatabase: ∆ x , ∆ x , ... ∆ y , ∆ y , ... ∆ x t ∆ y t MemoryNeuronNetwork ∆ˆ x t ∆ˆ y t e t z (cid:57) ∆ˆ x t (cid:57) ∆ˆ y t (cid:57) (cid:80) Fig. 1: Spatio-temporal lookahead modelrepresentation. Occupancy grid (cid:57) base is another popular rep-resentation for the spatial context. These approaches modeltrajectories as a D sequence, which can be unstructured attimes due to the missing temporal information. Extraction ofthe temporal context is normally done by using RNNs. [18]propose a Bayesian fuzzy model to accurately estimate thetemporal dependencies. [19] and [20] also use ConvolutionalNeural Networks to encode the temporal context. To unifythe spatial and temporal contexts, [21] follows a simpleand effective approach, where both the contexts are encodedtogether, using a Multi-Layer Perceptron, which drasticallyimproves the prediction performance. [22] use a RNN basedencoder-decoder along with [23] to model the spatio-temporalcontext. For the process of predicting future trajectories dif-ferent variants of RNNs have been used. [24] use a standardLSTM network for trajectory prediction on highways. [25] and[26] use Imitation Learning along with Generative AdversarialNetworks to predict future trajectories. [27] use LSTMs alongwith Convolutional Neural Networks with social pooling lay-ers and generate a multi-modal Gaussian model for trajectoryprediction. While these approaches have helped in improvingthe performance, they require heavy computational resources.This can make them quite hard to be implemented in real (cid:57) timescenarios.III. T RAJECTORY P REDICTION F RAMEWORK
Fig. 1 shows the proposed model for trajectory prediction.The figure consists of a trajectory database , that consists of allthe change in trajectory samples for multiple vehicles, presentin the dataset, and the Memory Neuron Network which isshown as a black box. At every time instant t , the trajectorydatabase provides the change in the ( x, y ) coordinates for aparticular vehicle, and the network estimates the next changein position of the vehicle. The initial values provided bythe trajectory database is fed to the network multiple timessequentially, so that the predicted values reach a steady (cid:57) state.Once the steady (cid:57) state is achieved, the network then receivesconsecutive input values from the trajectory database. t x t Fig. 2: The coordinate system is shown for a particular egovehicle in a multi-lane traffic environment. The y-axis is alongthe longitudinal direction and the x-axis is perpendicular to it.
A. Problem Formulation
The coordinate system used for formulating the trajectoryprediction problem is shown in Fig. 2. It shows the egovehicle (filled rectangle) and the non (cid:57) ego vehicles surroundingit (hollow rectangles). The location of the vehicle is measuredat its centre of mass in the local coordinate frame instead ofthe global coordinate frame (GPS data). The ego vehicle isassumed to be equipped with sensors that can measure theposition and velocity of the surrounding non (cid:57) ego vehicles inthe local coordinate frame . In this manner, it is possible toobtain the track histories of the non (cid:57) ego vehicles present inthe vicinity of the ego vehicle.The inherent uncertainties of the sensors only provide anapproximate estimate of the position and velocities of thesurrounding vehicles. As a result, it is challenging to predictthe future trajectories of these vehicles using simple kinematicequations. Thus, as followed in [28], a data - driven model isdeveloped that can relate the past track histories of the vehiclesto their future trajectories. As the values of the trajectory datacan change drastically when driving from one point to anotherover long periods of time, the difference between consecutive ( x, y ) coordinates are taken: ∆ x t = x t (cid:57) x t (cid:57) (1)where x t = ( x t , y t ) are the local coordinates of a vehicle attime instant t . As the datasets are generated through samplingdata points uniformly, the difference in the trajectory sampleswill be bounded within certain limit, ensuring network stabilityand improved performance. The trajectory prediction problemis then, posed as a system identification problem, with thestate of the system given by ∆ x t . Assuming this system is observable , from [29] the state of the system can be formulatedas: ∆ x t = F (∆ x t (cid:57) , ∆ x t (cid:57) , .. ) (2)where F ( . ) is an unknown nonlinear function of the previousstates. The goal of the network is to predict the next changein coordinates ( ∆ˆ x t ) of the vehicle at time t such that the cost z (cid:57) z (cid:57) α i (cid:57) α i Network neuronMemory neuron ∆ˆ x t (cid:57) ∆ˆ y t (cid:57) ... ∆ˆ x t ∆ˆ y tn i v i w i f i β L Fig. 3: The memory neuron network is fully connected with 6hidden neurons. Every neuron has a memory neuron associatedwith it. Initially, the network is trained with zero inputs sothat the weights stabilize to some equilibrium point, beforeproviding the actual data.function J is minimized at every time step. Here J is givenby J = (cid:107) ∆ x t (cid:57) ∆ˆ x t (cid:107) (3)where (cid:107) . (cid:107) represents the L norm. B. Network Architecture
The network architecture is shown in Fig. 3. The figureshows some of the network parameters that provides clarityon understanding the functioning of the network. The MemoryNeuron Network consists of fully connected network neurons (large open circles) and its associated memory neurons (smallfilled circles). There are weights associated with both theconnections of network neurons and memory neurons. Boththese weights are updated during backpropagation.To describe the functioning of the network, let ∆ x t (cid:57) =(∆ˆ x t − , ∆ˆ y t − ) be the inputs to the network. The net output n hj ( t ) of the j th network neuron in the hidden layer h can becalculated as: m hj ( t ) = (cid:88) k =1 w ikj n ik ( t ) + (cid:88) k =1 f ikj v ik ( t ) (4) n hj ( t ) = g h (cid:0) m hj ( t ) (cid:1) , ≤ j ≤ (5)where, • w ikj is the weight of the connection from k th networkneuron in the input layer i to j th network neuron of thehidden layer h . • n ik ( t ) is the output of the k th network neuron in the inputlayer i . In our case, n i ( t ) = ∆ˆ x t − and n i ( t ) = ∆ˆ y t − . f ikj is the weight of the connection from the memoryneuron corresponding to the k th network neuron in theinput layer i to j th network neuron of the hidden layer h . • v ik ( t ) is the output of the memory neuron of the k th network neuron in the input layer i . • g h ( . ) = tanh ( . ) is the activation function of the networkneurons present in the hidden layer.The output of the memory neuron corresponding to the j th network neuron in the layer l is given by: v lj ( t ) = α lj n lj ( t (cid:57)
1) + (1 (cid:57) α lj ) v lj ( t (cid:57) , l ∈ { i, h, L } (6)where α lj is the weight of the connection from j th networkneuron in the input layer l to its corresponding memoryneuron. The net output n Lj ( t ) of the j th network neuron inthe last layer L is calculated as: m Lj ( t ) = (cid:88) k =1 w hkj n hk ( t ) + (cid:88) k =1 f hkj v hk ( t ) + β Lj v Lj ( t ) (7) n Lj ( t ) = g L (cid:0) m Lj ( t ) (cid:1) , ≤ j ≤ (8)where, • β Lj is the weight of the connection from the memoryneuron to its corresponding j th network neuron in thelast layer L . • v Lj ( t ) is the output of the memory neuron correspondingto the j th network neuron in the last layer L . • g L ( . ) is a linear activation function with unit slope forthe network neurons in the output layer L . • n Lj ( t ) is the output of the j th network neuron in the lastlayer L . In our case, n L ( t ) = ∆ˆ x t and n L ( t ) = ∆ˆ y t .To ensure the stability of the network dynamics, the followingcondition is imposed: ≤ α lj , β Lj ≤ .The backpropagation algorithm is used to update all theweights of the network corresponding to both the networkneurons as well as the memory neurons. The following squarederror function is used for backpropagation: e ( t ) = (cid:88) j =1 ( n Lj ( t ) − d j ( t )) (9)where d j ( t ) is the desired teaching signal that is derived fromthe trajectory database. In our case, d ( t ) = ∆ x t and d ( t ) =∆ y t .At the time of updation t = τ , the weights are updated byusing the following rule: w lkj ( τ + 1) = w lkj ( τ ) − ηe l +1 j ( τ ) n li ( τ ) , l ∈ { i, h } (10) f lkj ( τ + 1) = f lkj ( τ ) − ηe l +1 j ( τ ) v li ( τ ) , l ∈ { i, h } (11)where η is the learning rate for the weights of the network,and e Lj ( τ ) = (cid:0) n Lj ( τ ) − d j ( τ ) (cid:1) , ≤ j ≤ (12) e hj ( τ ) = (cid:0) g h ( m ij ( τ )) (cid:1) (cid:48) (cid:88) p =1 e Lp ( τ ) w hjp ( τ ) , ≤ j ≤ (13) The various memory coefficients are updated using thefollowing equations: α lj ( τ + 1) = α lj ( τ ) − η (cid:48) ∂e∂v lj ( τ ) ∂v lj ∂α lj ( τ ) (14) β Lj ( τ + 1) = β Lj ( τ ) − η (cid:48) e Lj ( τ ) v Lj ( τ ) (15)where η (cid:48) is the learning rate for updating the memory coeffi-cients, and ∂e∂v hj ( τ ) = N l +1 (cid:88) s =1 f hjs ( τ ) e Ls ( τ ) (16) ∂v lj ∂α lj ( τ ) = n lj ( τ (cid:57) (cid:57) v lj ( τ (cid:57) (17)where N l +1 is the number of network neurons in the layer nextto l . The memory coefficients are hard (cid:57) limited to [0 , if theyhappen to fall outside the range. For a detailed discussion onthe functioning of the network and additional details, pleaserefer to [3].A crucial requirement in system identification problems is todetermine how many previous inputs and outputs are to be fedback to the model to capture the generic nonlinear input-outputmapping of the model. The presence of the memory neuronsensures that this requirements is optimally learnt during thelearning process. Note that the output of the network dependson the previous inputs as well as its own outputs due to thepresence of memory neurons in the output layer. Thus, theestimated next state of the system ∆ˆ x k is given by: ∆ˆ x t = ˆ F (∆ˆ x t (cid:57) , ∆ˆ x t (cid:57) , ... ) (18)where ˆ F ( . ) is the nonlinear transformation represented bythe Memory Neuron Network. The predicted samples ∆ˆ x t depends on the previous inputs due to the presence of memoryneurons in the input and hidden layers, and it depends on itsown previous outputs due to the presence of memory neuronsin the output layer. Thus, the spatio (cid:57) temporal look (cid:57) aheadmodel represented by Fig. 1 is known as parallel identificationmodel [30]. C. Training and Implementation Details
The Trajectory database consists of differences betweenconsecutive trajectory samples, as given by equation (1).During the learning process, at every time step t the networkreceives the previous state information ∆ x t (cid:57) , and predictsthe estimated next state ∆ˆ x t . The actual state of the system ∆ x t is then used as a teaching signal , to backpropagate thesquared error (cid:107) ∆ x t (cid:57) ∆ˆ x t (cid:107) and update both the weightsassociated with the network neurons and the memory neurons.The network consists of six neurons in the hidden layer, with tanh(.) as it’s activation function, and linear activation functionin the output layer. The range of the activation function isadjusted according to the range of the state values of thesystem, to avoid clipping during the prediction phase. It’s slopeABLE I: Root Mean Square Error (RMSE) values (in meters) are reported over a prediction horizon of s for the NGSIMdataset. Time CV CV-GMM[31] GAIL-GRU[32] LSTM MATF [33] CS-LSTM[27] S-LSTM[34] UST [21] UST-180[21] MNN s .
73 0 .
66 0 .
69 0 .
68 0 .
67 0 .
61 0 .
65 0 .
58 0 . . s .
78 1 .
56 1 .
56 1 .
65 1 .
51 1 .
27 1 .
31 1 .
20 1 . . s .
13 2 .
75 2 .
75 2 .
91 2 .
51 2 .
09 2 .
16 1 .
96 1 . . s .
78 4 .
24 4 .
24 4 .
46 3 .
71 3 .
10 3 .
25 2 .
92 2 . . s .
68 5 .
99 5 .
99 6 .
27 5 .
12 4 .
37 4 .
55 4 .
12 3 . . Algorithm 1:
Training pseudocode
Input :
A list D = [ d i ] , i = 1 , , · · · , n , where eachelement is a set of differential trajectorydata d i = (cid:110) ∆ x ( i ) t (cid:111) = (cid:110) (∆ x ( i ) t , ∆ y ( i ) t ) (cid:111) Tt =1 for vehicle i , learning rates η, η (cid:48) , epochs ; Output :
Trained memory neuron model fortrajectory prediction;
Initialize:
Initialize the weights of the networkarbitrarily, except the memory coefficientswhich are initialized to zero.; foreach d i ∈ D dofor e ← to epochs doforeach ∆ x t ∈ d i do Compute output of the network usingfeedforward equations (4) - (8);Compute error for backpropagation usingequation (9)Update all the weights and the memorycoefficients using equations (10) - (17); end foreachend forend foreach is also adjusted to provide a linear relationship with unit slope,about the origin.The entire trajectory data is taken for a vehicle, and thedifference between consecutive trajectory samples is calculatedand stored in the trajectory database for every vehicle. Theywill be referred as differential trajectory samples . Each sampleis then presented to the network sequentially and is trainedusing backpropagation. One epoch is said to be completedwhen the last sample in the set of differential trajectoriessamples is presented and learnt. This procedure is repeated for1,00,000 epochs, for multiple vehicle trajectories. The learningrates for both type of weights is chosen to be × − .Algorithm 1 summarizes the training procedure. The entiremodel is implemented in Python using
NumPy library [35].IV. P
ERFORMANCE E VALUATION
In this section, the proposed model is evaluated on twodatasets, and the performance is compared quantitatively withseveral state (cid:57) of (cid:57) art techniques by employing the RMSE met-ric. A. Datasets
For evaluating the performance of the proposed model, thefollowing datasets are used:(a)
NGSIM US-101 [36] : The Next Generation Simulation(NGSIM) US (cid:57) dataset consists of trajectory datasampled at Hz, over a span of minutes. Thetrajectory data is reported in both global as well as localcoordinate frames. These trajectories are recorded froma fixed bird’s eye view, and consists of varying trafficconditions. A similar experimental setup is followed asin [27], where s of trajectory history is chosen topredict the estimated trajectories over the horizon of next s during the testing phase.(b) Synthetic Dataset:
In order to predict trajectories ofrogue vehicles, the trajectories for 20 different roguevehicles is generated by using the
CARLA simulator. Therogue vehicles are made to skip traffic lights randomlyand move in a zig (cid:57) zag fashion within the lane, whiletraveling at a dangerously high velocity. They can alsochange lanes abruptly without any indication. The trajec-tory data is sampled at Hz over a (cid:57) minute duration.In order to capture abrupt changes in the trajectoriesof rogue vehicles, they are sampled at a higher rate of Hz. The same procedure of choosing s of trajectoryhistory and predicting the estimated trajectories over thehorizon of next s during the testing phase is followed. B. Evaluation metric
During the prediction phase, the differential trajectory sam-ples from the trajectory database is provided for a duration of s to the network and for the next s , the input to the networkis it’s previous outputs. The predicted values of the networkare summed up with the starting actual trajectory values ofeach vehicle over the duration of s to generate the predictedactual trajectory of the vehicle. In order to compare the resultsof the proposed model quantitatively, the root mean squarederror (RMSE) metric is used over all future time steps T H andnumber of vehicles N :RMSE = (cid:80) Nn =1 (cid:114) (cid:80) THt =1 (cid:13)(cid:13)(cid:13) x ( n ) t − ˆ x ( n ) t (cid:13)(cid:13)(cid:13) T H N (19)ig. 4: Simulating trajectory prediction on CARLA for two rogue vehicles. The trained model is deployed on each of therogue vehicle present in the simulation, so that the other vehicles present in the traffic get a ‘ s ’ look (cid:57) ahead of every roguevehicle. This way, they can plan some protective measures to avoid any collision with them. The predicted trajectories areshown frame-by-frame in green dotted lines for future s , and the actual trajectories given by the planner are shown for s in red dotted lines. Figure on top shows a car traveling at a roundabout. The bottom figure shows the trajectory prediction ata junction. C. Results
The performance of the Memory Neuron Network is re-ported along with several state-of-the-art algorithms tested onthe NGSIM US-101 dataset in Table I. The table consistsof the RMSE for a look (cid:57) ahead duration of s to s for9 algorithms, which has been reproduced from [21]. It isevident that the Memory Neuron Network outperforms all theother algorithms. Our results have improved by for s prediction horizon, and about for s prediction horizonwhen compared to [21]. Further, the rise in the RMSE valuesfrom s horizon to s horizon is far less for our proposedmodel, as compared to other algorithms. From this analysis,it can also be concluded that the proposed model is relativelymore stable, than the current existing algorithms.This superior performance can be attributed to the factthat the memory neurons not only remember their own pastvalues, but the past values of all the other memory neuronsin it’s preceding layers as well. This makes the MemoryNeuron Network globally recurrent , as compared to the LSTM networks which are locally recurrent.To test it’s robustness, the trained model is deployed in real-time simulation, with 100 cars. The simulation is carried outusing
C++ APIs provided by
CARLA ’ s unreal environment.Only the feedforward part of trained network is implementedin each of the rogue vehicle’s trajectory planner. About of them are rogue vehicles. The simulation consists of mixedvehicles, ranging from small cars to heavy trucks. The futuretrajectories of all the rogue vehicles are predicted, based ontheir current location and their s past track histories. Theprediction of the trajectories are shown for two different roguevehicles as frame-by-frame snapshots in Fig. 4.It can be observed from Fig. 4 that there is minimalerror between the predicted trajectories and the actual futuretrajectories, when the vehicle is travelling in a near (cid:57) straightpath. The bottom left figure shows the predicted trajectoriesat the beginning of a left-turn manoeuvre. It is evident thatthere is a relatively higher error in this scenario, as the model A detailed video demonstration on the same can be found here. annot anticipate the radius of curvature of the turning dueto the fact that it has no prior knowledge about the map andthe dimensions of the roads and junctions present in the map.This shouldn’t be concerning, as the predicted trajectory hasthe same structure of the actual future trajectory, and thus itcan be inferred that the vehicle is still going to take a left (cid:57) turn.V. C
ONCLUSIONS AND FUTURE WORKS
This paper presents a trajectory prediction model, whichuses a novel recurrent neural network as its base model. Thetrajectory prediction problem is posed as a system identifica-tion problem, where the Memory Neuron Network learns theinput-output relationship between the past trajectory samplesand the future predicted trajectory samples. It is clear that theproposed model outperformed all the state (cid:57) of (cid:57) art algorithmscurrently available, and is also very efficient in the sense thatit requires less resources when training, computationally fasterdue to it’s less complicated architecture. The proposed modelhas a RMSE that is about lesser than the RMSE reportedby the current state (cid:57) of (cid:57) art algorithms, for a s look (cid:57) aheadprediction . The robustness of the proposed model is alsoverified by deploying it in the CARLA simulator, for each roguevehicle. While the model performs very well in relativelystraighter paths, it fails to predict the trajectories accurately ata junction as it is not aware of the structure of the map. Theproposed model will be improved in this regards by addingsome features related to the roads and junctions present in themap during the training process, in one of our future works.VI. A
CKNOWLEDGMENTS
The authors would like to thank Dr. Shirin Dora and Dr.Chandan Gautam for their valuable suggestions and comments,and would also like to acknowledge the Wipro (cid:57)
IISc ResearchInnovation Network (WIRIN) for their financial support.R
EFERENCES[1] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
Neuralcomputation , vol. 9, no. 8, pp. 1735–1780, 1997.[2] K. Cho, B. Van Merri¨enboer, C. Gulcehre, D. Bahdanau, F. Bougares,H. Schwenk, and Y. Bengio, “Learning phrase representations usingrnn encoder-decoder for statistical machine translation,” arXiv preprintarXiv:1406.1078 , 2014.[3] P. Sastry, G. Santharam, and K. Unnikrishnan, “Memory neuron net-works for identification and control of dynamical systems,”
IEEEtransactions on neural networks , vol. 5, no. 2, pp. 306–319, 1994.[4] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun,“CARLA: An open urban driving simulator,” in
Proceedings of the 1stAnnual Conference on Robot Learning , 2017, pp. 1–16.[5] G. Xie, H. Gao, L. Qian, B. Huang, K. Li, and J. Wang, “Vehicle trajec-tory prediction by integrating physics-and maneuver-based approachesusing interactive multiple models,”
IEEE Transactions on IndustrialElectronics , vol. 65, no. 7, pp. 5999–6008, 2017.[6] H. Veeraraghavan, N. Papanikolopoulos, and P. Schrater, “Deterministicsampling-based switching kalman filtering for vehicle tracking,” in . IEEE, 2006, pp.1340–1345.[7] R. Schubert, E. Richter, and G. Wanielik, “Comparison and evaluationof advanced motion models for vehicle tracking,” in . IEEE, 2008, pp. 1–6.[8] L. Li, Y. Liu, J. Wang, W. Deng, and H. Oh, “Human dynamics baseddriver model for autonomous car,”
IET Intelligent Transport Systems ,vol. 10, no. 8, pp. 545–554, 2016. [9] A. L. Ferreira, G. F. B. Piccinini, S. Rˆola, and A. Sim˜oes, “Genderand age-related differences in the perception of in-vehicle mobile phoneusage among portuguese drivers,”
IET Intelligent Transport Systems ,vol. 7, no. 2, pp. 223–229, 2013.[10] M. L. Puterman,
Markov decision processes: discrete stochastic dynamicprogramming . John Wiley & Sons, 2014.[11] X. Zou and D. M. Levinson, “Modeling pipeline driving behaviors:Hidden markov model approach,”
Transportation research record , vol.1980, no. 1, pp. 16–23, 2006.[12] P. Boyraz, A. Sathyanarayana, J. H. Hansen, and E. Jonsson, “Driver be-havior modeling using hybrid dynamic systems for ‘driver-aware’activevehicle safety,”
Proceedings of the Enhanced Safety of Vehicles , pp. 1–8,2009.[13] N. Dapzol, “Driver’s behaviour modelling using the hidden markovmodel formalism,” in
ECTRI Young researchers seminar, The Hague,the Netherlands , vol. 2, no. 2.2, 2005, pp. 2–1.[14] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, “Humanbehavior modeling with maximum entropy inverse optimal control.” in
AAAI Spring Symposium: Human Behavior Modeling , vol. 92, 2009.[15] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social gan:Socially acceptable trajectories with generative adversarial networks,” in
Proceedings of the IEEE Conference on Computer Vision and PatternRecognition , 2018, pp. 2255–2264.[16] A. Vemula, K. Muelling, and J. Oh, “Social attention: Modeling attentionin human crowds,” in . IEEE, 2018, pp. 1–7.[17] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning onpoint sets for 3d classification and segmentation,” in
Proceedings of theIEEE conference on computer vision and pattern recognition , 2017, pp.652–660.[18] S. Subhrajit, P. Mahardhika, and S. Sundaram, “Bayesian neuro-fuzzyinference system (banfis) for temporal dependency estimation,”
IEEETransactions on Fuzzy Systems , 2020.[19] X. Li, X. Ying, and M. C. Chuah, “Grip: Graph-based interaction-awaretrajectory prediction,” in . IEEE, 2019, pp. 3960–3966.[20] N. Nikhil and B. Tran Morris, “Convolutional neural network fortrajectory prediction,” in
Proceedings of the European Conference onComputer Vision (ECCV) , 2018, pp. 0–0.[21] H. He, H. Dai, and N. Wang, “Ust: Unifying spatio-temporal con-text for trajectory prediction in autonomous driving,” arXiv preprintarXiv:2005.02790 , 2020.[22] K. Messaoud, I. Yahiaoui, A. Verroust, and F. Nashashibi, “Attentionbased vehicle trajectory prediction,”
IEEE Transactions on IntelligentVehicles , 2020.[23] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation byjointly learning to align and translate,” arXiv preprint arXiv:1409.0473 ,2014.[24] F. Altch´e and A. de La Fortelle, “An lstm network for highway trajectoryprediction,” in . IEEE, 2017, pp. 353–359.[25] R. P. Bhattacharyya, D. J. Phillips, B. Wulfe, J. Morton, A. Kuefler,and M. J. Kochenderfer, “Multi-agent imitation learning for drivingsimulation,” in . IEEE, 2018, pp. 1534–1539.[26] W. Si, T. Wei, and C. Liu, “Agen: Adaptable generative predictionnetworks for autonomous driving,” in . IEEE, 2019, pp. 281–286.[27] N. Deo and M. M. Trivedi, “Convolutional social pooling for vehicletrajectory prediction,” in
Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition Workshops , 2018, pp. 1468–1476.[28] M. V. Kumar, S. Omkar, R. Ganguli, P. Sampath, and S. Suresh,“Identification of helicopter dynamics using recurrent neural networksand flight data,”
Journal of the American Helicopter Society , vol. 51,no. 2, pp. 164–174, 2006.[29] I. Leontaritis and S. A. Billings, “Input-output parametric models fornon-linear systems part i: deterministic non-linear systems,”
Interna-tional journal of control , vol. 41, no. 2, pp. 303–328, 1985.[30] K. Narendra and K. Parthasarathy, “Identification and control of dy-namical systems containing neural networks,”
IEEE Trans. on Neuralnetworks. , vol. 2, no. 2, pp. 252–262, 1991.[31] N. Deo, A. Rangesh, and M. M. Trivedi, “How would surround vehiclesmove? a unified framework for maneuver classification and motionrediction,”
IEEE Transactions on Intelligent Vehicles , vol. 3, no. 2,pp. 129–140, 2018.[32] A. Kuefler, J. Morton, T. Wheeler, and M. Kochenderfer, “Imitatingdriver behavior with generative adversarial networks,” in . IEEE, 2017, pp. 204–211.[33] T. Zhao, Y. Xu, M. Monfort, W. Choi, C. Baker, Y. Zhao, Y. Wang,and Y. N. Wu, “Multi-agent tensor fusion for contextual trajectoryprediction,” in
Proceedings of the IEEE Conference on Computer Visionand Pattern Recognition , 2019, pp. 12 126–12 134.[34] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, andS. Savarese, “Social lstm: Human trajectory prediction in crowdedspaces,” in
Proceedings of the IEEE conference on computer vision andpattern recognition , 2016, pp. 961–971.[35] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers,P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J.Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett,A. Haldane, J. F. del R’ıo, M. Wiebe, P. Peterson, P. G’erard-Marchant,K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke,and T. E. Oliphant, “Array programming with NumPy,”
Nature ,vol. 585, no. 7825, pp. 357–362, Sep. 2020. [Online]. Available:https://doi.org/10.1038/s41586-020-2649-2[36] J. Colyar and J. Halkias, “Us highway 101 dataset,”
Federal HighwayAdministration (FHWA), Tech. Rep. FHWA-HRT-07-030 , 2007.[37] I. Ajzen et al. , “The theory of planned behavior,”
Organizationalbehavior and human decision processes , vol. 50, no. 2, pp. 179–211,1991.[38] V. Kosaraju, A. Sadeghian, R. Mart´ın-Mart´ın, I. Reid, H. Rezatofighi,and S. Savarese, “Social-bigat: Multimodal trajectory forecasting usingbicycle-gan and graph attention networks,” in