Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction
Ajay Jain, Sergio Casas, Renjie Liao, Yuwen Xiong, Song Feng, Sean Segal, Raquel Urtasun
DDiscrete Residual Flow for ProbabilisticPedestrian Behavior Prediction
Ajay Jain ∗ † , Sergio Casas ∗ , Renjie Liao ∗ , Yuwen Xiong ∗ , Song Feng , Sean Segal , Raquel Urtasun Uber Advanced Technologies Group , University of Toronto , UC Berkeley [email protected] , { sergio.casas,rjliao,yuwen,songf,ssegal,urtasun } @uber.com Abstract:
Self-driving vehicles plan around both static and dynamic objects, ap-plying predictive models of behavior to estimate future locations of the objectsin the environment. However, future behavior is inherently uncertain, and modelsof motion that produce deterministic outputs are limited to short timescales. Par-ticularly difficult is the prediction of human behavior. In this work, we proposethe discrete residual flow network (DRF-N ET ), a convolutional neural network forhuman motion prediction that captures the uncertainty inherent in long-range mo-tion forecasting. In particular, our learned network effectively captures multimodalposteriors over future human motion by predicting and updating a discretizeddistribution over spatial locations. We compare our model against several strongcompetitors and show that our model outperforms all baselines. Keywords:
Deep Learning, Autonomous Driving, Uncertainty, Forecasting
In order to plan a safe maneuver, a self-driving vehicle must predict the future motion of surroundingvehicles and pedestrians. Motion prediction is challenging in realistic city environments. In Figure 1,we illustrate several challenges for pedestrian prediction. Gaussian distributions often poorly fitstate posteriors (Fig. 1-a). Further, pedestrians have inherently multimodal behavior, as they canmove in arbitrary directions and have unknown and changing goals, each achievable with multipletrajectories (Fig. 1-b). Even with strong evidence for a particular action, such as a road crossing,partially observed environments increase uncertainty in the timing of the action (Fig. 1-c). However,a self-driving vehicle motion planner needs actor predictions to be associated with time. Additionalchallenges include efficiently integrating spatial and temporal information, the mixed continuous-discrete nature of trajectories and maps, and availability of realistic data.In the context of self-driving, most prior work represents behaviors through trajectories. Futuretrajectories can be predicted with a recurrent neural network (RNN) [1, 2, 3], a convolutional neuralnetwork (CNN) [4, 5, 6], or with constant velocity, constant acceleration, or expert-designed heuristics.However, a trajectory that minimizes the mean-squared error with respect to the true path can onlycapture the conditional average of the posterior [7]. The conditional average trajectory does notrepresent all possible future behaviors and may even be infeasible, lying between feasible trajectories.To express multiple possible behaviors, a fixed number of future trajectories can be predicted [8],or several can be sampled [3, 9]. Still, in realistic environments, posterior predictive distributionsare complex and a large number of samples are needed to capture the space of possibilities. Suchmodels tradeoff prediction completeness and latency from repeated sampling. Further, the number ofpossible trajectories increases exponentially over long time horizons, and uncertainty grows rapidly.Instead of predicting trajectories, in this work, we take a probabilistic approach, predicting distri-butions over pedestrian state at each timestep that can directly be used for cost-based self-drivingvehicle planning. Conditioning on a spatio-temporal rasterization of agent histories aligned to thelocal map, we leverage deep convolutional neural network architectures for implicit multi-agent rea-soning, and mimic human dynamics through a discrete residual flow network , which we refer to asDRF-N ET . We summarize our contributions as follows: ∗ Denotes equal contribution. † Work done while at Uber ATG.3rd Conference on Robot Learning (CoRL 2019), Osaka, Japan. a r X i v : . [ c s . C V ] O c t ≤ (Ω) ? (a) (c) t=1t=2t=3 (b) p x t ( · | Ω) t=0 Figure 1: Challenging urban scenarios for pedestrian prediction, depicting pedestrian detections(circles) and future state posteriors colored by time horizon. (a) Gaussian distributions often poorlyexpress scene-sensitive behaviors. (b) Inherent multimodality: the pedestrian may cross a crosswalkor continue along a sidewalk. (c) Partial observability: signals and actors may be occluded. • We develop a deep probabilistic formulation of actor motion prediction that providesmarginal distributions over state at each future timestep without expensive marginalizationor sampling. Our discrete residual flow equation is motivated by autoregressive generativemodels, and better captures temporal dependencies than time-independent baselines. • We propose the convolutional Discrete Residual Flow Network that predicts actor state overlong time horizons with highly expressive discretized distributions. • We thoroughly benchmark model variants and demonstrate the benefit of belief discretiza-tion on a large scale, real-world dataset. We evaluate the likelihood , displacement error , multimodality , entropy , semantic mass ratio and calibration of the predictions, using anovel ModePool operator for estimating the number of modes of a discrete distribution. Prior work on pedestrian prediction has largely modeled trajectories, goals, or high-level intent.
Human trajectory forecasting
The pedestrian prediction literature is reviewed in [10, 11]. Multi-pedestrian interactions have been modeled via pooling [1, 3] or game theory [12]. Becker et al. [2]predict future trajectories with a recurrent encoder and MLP decoder, reporting lower error than moreelaborate multi-agent schemes, and find that behaviors are multimodal and strongly influenced by thescene. Social GAN [3] is a sequence-to-sequence generative model where trajectory samples vary inspeed and turning angle, trained with a variety loss to encourage diversity. However, the runtime ofthe sampling approach scales with the number of samples ( ms for trajectories), even withoutusing a local map, and many samples are needed. SoPhie [9] is another sampling strategy integratingexternal overhead camera imagery. In contrast, we predict entire expressive spatial distributions ratherthan individual samples and incorporate a local map into prediction. Goal directed prediction
Ziebart et al. [13] use historical paths to pre-compute a prior distributionover pedestrian goals indoors, then develop an MDP to infer a posterior distribution over futuretrajectories. Wu et al. [14] use a heuristic to identify possible goal locations in a mapped environmentand a Markov chain to predict the next-time occupancy grid. Rehder et al. [15, 16] use a two-stagedeep model to predict a Gaussian mixture over goals, then construct distributions at intermediatetimesteps with a planning network. Still, the number of mixture components must be tuned, and themixture is discretized during inference, which is computationally expensive. Fisac and Bajcsy [17, 18]specify known goals for each human indoors, then estimate unimodal state distributions by assuminghumans approximately maximize utility i.e . progress toward the goal measured by Euclidean norm.They estimate prediction confidence from model performance and return uninformative distributionsat low confidence. Confidence estimation is complementary to our approach.
Semantic map
Pedestrian predictors have separately reasoned about spatially continuous trajecto-ries and discretized world representations [13, 15]. These works either ignore the semantic map orintegrate it at an intermediate stage. In vehicle prediction, input map rasterizations are more widelyused. IntentNet [5] renders a bird’s-eye view of the world to predict vehicle trajectories and high-level intention simultaneously, using a rasterized lane graph and a 2D convolutional architecture toimprove over previous work [4]. Similar map rasterizations are used in [6, 19, 20], and this work.
Related modeling techniques
The convolutional long short-term memory (ConvLSTM) architec-ture has been applied to spatio-temporal weather forecasting [21]. A ConvLSTM iteratively updatesa hidden feature map, from which outputs are derived. In contrast, DRF-N ET sequentially adapts theoutput space rather than a hidden state. Similarly, the adaptive instance normalization operator [22]uses a shared feature to predict and apply scale/shift parameters to a fixed, discrete image. Normaliz-ing flows [23] apply a series of invertible mappings to samples from a simple prior, e.g . a Gaussian,constructing a random variable with a complex PDF. While normalizing flows transform individualsamples, we directly transform a probability mass function (PMF) for computational efficiency.2 iscreteResidual Flow step + ++ + Residuallog prob
Negative log likelihood L L L p x t − ( · | Ω) p x t +1 ( · | Ω) p x t ( · | Ω) FPN
Backbone Feature Pyramid Network
Scene embedding Discrete Residual Flow head Beliefs over future statesScene rasterization
MapTraffic lightsDynamic objects … Figure 2: Overview of the Discrete Residual Flow Network. Pedestrian of Interest (PoI) and actordetections are aligned with a semantic map. A multi-scale backbone jointly reasons over spatio-temporal information in the input, embedding context into a feature F . Finally, the DRF headrecursively adapts an initial distribution to predict future pedestrian states on long time horizons. In this paper, we express beliefs over future pedestrian positions through categorical distributionsthat discretize space. Such distributions can be used for cost-based planning or constrained pathoptimization in self-driving vehicles. In this section, we explain how we represent historical observa-tions as a multi-channel image encoding both the known map and detected actors, a process we call rasterization . We then introduce a backbone deep neural network which extracts features from therasterized image, followed by the probabilistic framework for our DRF-N ET . Finally, we introduceour DRF head which uses the extracted features for prediction. Encoding Historical Information
Future pedestrian actions are highly correlated with historicalactions. However, actions are also influenced by factors such as road surface types, traffic signals,static objects, vehicles, and other pedestrians. We rasterize all semantic map information and agentobservations into a 3D tensor, encoding both spatial and temporal information by automatic rendering.The first two dimensions correspond to the spatial domain and the third dimension forms channels.Each channel is an × px image encoding specific local bird’s eye view (BEV) informationat a resolution of px per meter. Figure 3 shows an example rasterization from a real urban scene.Dynamic agents are detected from LiDAR and camera with the object detector proposed in Lianget al. [24], and are associated over time using a matching algorithm. Resulting trajectories are refinedusing an Unscented Kalman Filter [25]. DRF-N ET renders detected pedestrians in each timestep t for the past seconds in channel D t and detected non-pedestrians ( e.g . vehicles) in channel V t . Todiscriminate the pedestrian of interest from other actors, a grayscale image R masks their tracklet.DRF-N ET renders the local map in a similar fashion to [5], though centers the map about thePoI. semantic map channels M finely differentiate urban surface labels. These channels maskcrosswalks, drivable surfaces, traffic light state, signage, and detailed lane information. Maps areannotated in a semi-automated fashion in cities where the self-driving vehicle may operate, andonly polygons and polylines are stored. The final rasterization is Ω = [ D ≤ , V ≤ , R, M ] where [ · ] indicates concatenation along the channel dimension, the subscript ≤ indicates a collection ofelements from all past timesteps and t = 0 is the last timestep. All channels are rotated such that thecurrently observed PoI is oriented toward the top of the scene. Backbone Network
DRF-N ET uses a deep residual network with convolutional layers (ResNet-18) [26] to extract features F from the rasterization Ω . We extract feature maps at , , and of the input resolution from ResNet-18. These multi-scale intermediate features are upscaled andaggregated into a resolution global context with a feature pyramid network (FPN) [27]. Probabilistic Actor State Prediction
We now introduce a probabilistic formulation of future actorstate prediction. Given rasterization Ω , we are interested in inferring a predictive posterior distributionover possible spatial locations of the PoI for each timestep t where t = 1 , · · · , T f . Instead of treatingthe state as a continuous random variable, we discretize space to permit a one-hot state encoding.Specifically, we divide space into a grid with K bins. The state at time t , x t , is a discrete randomvariable which takes one of the K possible bins.Consider the joint probability of the states in the future T f timesteps, i.e ., p x ··· , x Tf ( x , · · · , x T f | Ω) .This distribution can be modeled with several factorizations. The first and the most straightforwardfactorization assumes conditional independence of future timesteps, p x , ··· , x Tf ( x , · · · , x T f | Ω) = (cid:81) t p x t ( x t | Ω) (1)3 b) Semantic map channels Crossings, road mask, lights, lanes (a) Aggregated rasterization
Dynamic objects shown at t=0 (c) PoI + dynamic object history
Subset of timesteps shown
Figure 3:
Scene history and context representation.
DRF-N ET rasterizes map elements into ashared spatial representation (b), augmented with spatio-temporal encodings of actor motion (c).We can use a neural network, e.g ., a CNN, to directly model p x t ( x t | Ω) . In Section 4.3, we showthe performance of a mixture density network and fully-convolutional predictor that simultaneouslypredict these factors. Still, conditional independence is a strong assumption. The second factorizationfollows an autoregressive fashion, providing the foundation for many models in the literature, p x , ··· , x Tf ( x , · · · , x T f | Ω) = (cid:81) t p x t | x ≤ t − ( x t | x ≤ t − , Ω) (2)For example, recurrent encoder-decoder architectures [1, 2, 3] sample trajectories one state at a timeand capture the conditional dependencies through a hidden state.In contrast to the sample-based approach, often we desire access to compact representations of p x t ( x t | Ω) for a particular Ω , such as an analytic form or a discrete categorical distribution. As wealways condition on Ω , we refer to p x t ( x t | Ω) as a marginal distribution. Access to the marginalprovides interpretability, parallel sampling and ease of planning as the marginals can be used asoccupancy grids. However, direct marginalization is expensive if not intractable as we typicallyhave no simple analytic form of the joint distributions. Approximation is possible with Monte Carlomethods, though many samples are needed to characterize the marginal.Instead, we propose a flow between marginal distributions that resembles an autoregressive model inits iterative nature, but avoids sampling at each step. In contrast to a normalizing flow [23], whichapproximates a posterior over a single random variable by iteratively transforming its distribution, dis-crete residual flow transforms between the marginal distributions of different, temporally correlatedrandom variables by exploiting a shared domain. Discrete Residual Flow
Our model recursively constructs p x t ( · | Ω) from p x t − ( · | Ω) , log p x t ( x t | Ω) = log p x t − ( x t | Ω) + log ψ t ; θ t ( x t , p x t − ( · | Ω) , Ω) (cid:124) (cid:123)(cid:122) (cid:125) Residual − log Z t , (3)where we refer to the second term on the right hand side as the residual . ψ t ; θ t is a sub-networkwith parameter θ t called the residual predictor that takes the marginal distribution p x t − ( · | Ω) and context Ω as input, and predicts an elementwise update that is used to construct the subsequentmarginal distribution p x t ( · | Ω) . Z t is the normalization constant to ensure p x t ( · | Ω) is a validdistribution. Note that the residual itself is not necessarily a valid probability distribution.Eq. (3) can be viewed as a discrete probability flow which maps from the distribution of x t − tothe one of x t . We use deep neural networks to instantiate the probability distributions under thisframework and provide a derivation of Eq. (3) in the appendix, Section 6.5.For initialization, p x ( · | Ω) is constructed with high value around our t = 0 PoI position and near-zero value over other states. In implementation, the residual predictor is a convolutional architecturethat outputs a 2D image, a compact and convenient representation as our states are spatial. This2D image is queryable at state x t via indexing, as is the updated marginal. Additionally, in imple-mentation, we normalize all marginals at once and apply residuals to the unnormalized potential ˜ p , log ˜ p x t ( x t | Ω) = log ˜ p x t − ( x t | Ω) + log ψ t ; θ t ( x t , ˜ p x t − ( · | Ω) , Ω) (4)Figure 2 illustrates the overall computation process. The embedding of the rasterization F (Ω) isshared at all timesteps, used by each residual predictor. Figure 4 further illustrates the architecturaldetails of the DRF residual predictor for one timestep. Learning
We perform learning by minimizing the negative log likelihood (NLL) of the observedsequences of pedestrian movement. Specifically, we solve the following optimization, min Θ − E x , Ω (cid:104)(cid:80) T f t =1 log p x t ( x t | Ω) (cid:105) (5)4 og ˜ p x t ( · | Ω)log ˜ p x t − ( · | Ω) log p x t ( · | Ω)
128 128 1286464 log ψ t ; θ t Figure 4: One step of recursive Discrete Residual Flow. The log potential is used to update the globalfeature map F . DRF then predicts a residual ψ t ; θ t to flow to the log potential for the next timestep. Negative log likelihood (NLL) ADE (m) FDE (m) Mass Ratio (%)Model Mean @ 1 s @ 3 s @ 10 s 0.2-10s @ 1 s @ 3 s @ 10 s Acc. RecallDensity Net 5.39 2.87 3.96 6.74 3.49 0.93 1.72 7.66 77.99 81.33MDN-4 3.01 1.64 2.00 4.33 1.47 0.38 0.69 3.38 87.85 84.12MDN-8 3.43 1.60 2.77 4.79 1.78 0.60 0.88 3.91 85.56 84.19ConvLSTM 2.51 0.89 1.86 4.07 1.58 0.47 1.06 3.20 88.02 85.02DRF-N ET Table 1: Comparison of the baselines and our proposed model DRF-N ET with access to ground-truthobservations. Metrics are negative log likelihood in . × . m bin containing future GT position,average displacement error (ADE) and final displacement error (FDE) in meters, and percent ofpredicted mass. Mean NLL, ADE and the mass ratios are averaged over 50 timesteps, t = 0 . − s.where the expectation E [ · ] is taken over all possible sequences and will be approximated via mini-batches. Θ = { θ , · · · , θ T f , w } where w denotes the parameters of the backbone network. There is not a standard dataset for probabilistic pedestrian prediction with real-world maps anddynamic objects. Thus, we construct a large-scale dataset of real world recordings, object annotations,and online detection-based tracks. We implement baseline pedestrian prediction networks inspiredby prior literature [28, 29, 15] and compare DRF-N ET against these baselines on standard negativelog likelihood and displacement error measures. We propose an evaluation metric for measuringprediction multimodality, which is one of the most characteristic properties of pedestrian behavior.We also analyze the calibration, entropy and semantic interpretation of predictions. Finally, we presentqualitative results in complex urban scenarios. Our dataset consists of 481,927 ground truth pedestrian trajectories gathered in several North-American cities. The dataset is split into 375,700 trajectories for training, 34,571 for validation,and 71,656 held-out trajectories for testing. Dynamic objects are manually annotated in a ◦ , m range view from an on-vehicle LiDAR sensor. Annotations contain s ( frames) of pastobservations and s ( frames) of the future. These Hz, s sliding windows are extracted fromlonger logs.We also fine-tune and evaluate DRF-N ET with variable length trajectories from an object detectorin the same scenarios. The detector is discussed in Section 3. This assesses real-world, on-vehicleprediction performance, reflecting the challenges inherent to real perception such as partial observ-ability, occlusion and identity switches in tracking algorithms. While PoIs are annotated for a full seconds in our ground truth experiments, realistic tracks are of variable length. A self-drivingvehicle must predict the behavior of other agents with a very limited set of observations. Thus, weevaluate DRF-N ET by predicting seconds ( frames) into the future, given tracks with as fewas historical frames, sufficient for estimating acceleration. Relaxing the requirements about pasthistory avoids skewing our dataset toward easily tracked pedestrians, such as stationary agents. In this section, we describe two baseline predictor families. These baselines are trained end-to-endto predict distributions given features F (Ω) produced by the same backbone as our proposed model.5 eal detection data (NLL)Model Mean @ 1 s @ 3 s @ 10 sDensity Net 5.64 1.88 4.12 7.91MDN-4 3.21 1.52 2.54 4.71MDN-8 3.21 1.53 2.55 4.73ConvLSTM 3.14 1.54 2.51 4.64DRF-N ET Table 2: Probabilistic prediction comparison of the baselinesand our proposed model DRF-N ET when noisy detections(online tracks) are observed instead of the ground-truth. MDN-4(2.3)ConvLSTM (1.4)DRF-Net (1.1) A cc u r acy Model confidence
Figure 5: Calibration curves and ex-pected calibration error ( ∗ − % ) Mixture Density Networks (MDNs) represent a conditional posterior over continuous targets givencontinuous inputs with a fully-connected neural network that predicts parameters of Gaussian mixturemodel [7]. For a baseline, we implement a variant of this architecture that models pedestrian posteriorsat multiple time horizons, conditioned on the past history and current location. Inspired by Rehderet al. [15], we generate the i -th mixture component from the neuron outputs { m x , m y , s x , s y , r, p } i which are then reparameterized as σ x,i = exp ( s x,i ) + (cid:15), σ y,i = exp ( s y,i ) + (cid:15), and ρ i = tanh ( r i ) to obtain the mean (cid:126)µ i , covariance matrix Σ i and the responsibility of the mixture π i : (cid:126)µ i = (cid:20) m x,i m y,i (cid:21) , Σ i = (cid:20) σ x,i ρ i σ x,i σ y,i ρ i σ x,i σ y,i σ y,i (cid:21) , π i = exp ( p i ) (cid:80) Nj =1 exp ( p j ) (6)Training MDNs is challenging due to a high sensitivity to initialization and parameterization. Toavoid numerical instabilities, the minimum standard deviation is (cid:15) . Even with a careful initializationand parameterization, training can be unstable, which we mitigate by discarding abnormally largelosses. Note that Rehder et al. [15] stabilized training by minimizing only the minimum of thebatchwise negative log likelihood. Minimizing this minimum loss leads to a good performance oneasy examples, but catastrophic performance on hard ones. Lastly, conversions from a discretizedspatial input to a continuous output can be challenging to learn [30], a problem that our proposedDRF-N ET avoids via a discretized output that is spatially aligned with the input. ConvLSTM
In contrast to our DRF-N ET that recursively updates output distributions in the log-probability space, one can also recurrently update hidden state using a Convolutional LSTM [21]that observes the previous prediction. Output distributions are then predicted from the hidden state. We evaluate negative log likelihood (NLL) at short and long prediction horizons, where lower valuesindicate more accurate predictions, as well as the mean NLL across all 50 future timesteps. InTable 1 and 2, we present results on the held-out test set for ground truth annotated logs and tracked,real-world detections, respectively. Our proposed DRF-N ET achieves a superior likelihood over thebaselines by introducing a discrete state representation and a probability flow between timesteps. Likelihood on ground truth tracks
In order to evaluate our results under perfect perception, webenchmark on ground truth (annotated) pedestrian trajectories. Table 1 shows that our proposedmodel reduces the mean NLL by . when compared to the best performer among the MDNs andby . with respect to the ConvLSTM baseline. This corresponds to a increase in geometricmean likelihood compared to the best MDN and to a increase when compared to the ConvLSTM. Likelihood on online tracks
Under online, imperfect perception, DRF-N ET achieves a reductionof . in mean NLL over the best MDN and . over ConvLSTM, i.e . a and a increaseof the geometric mean likelihood of the future observed pedestrian positions, respectively (Table 2).DRF-N ET ’s sequential residual updates may regularize and smooth predictions despite perceptionnoise. Adding more than components to the density networks does not reduce NLL. Directlypredicting occupancy probability over a grid delivers stronger performance than discretizing a con-tinuous spatial density. Using an explicit memory with hidden state updates (ConvLSTM) also hasinferior performance to our proposed flow between output distributions. Displacement error
We compute the expected root mean squared error, or expected displacementerror, between the ground truth pedestrian position and model predictions. This is approximated bydiscretizing posteriors, computing the distance from each cell to the ground truth, and taking theaverage weighted by confidence at each cell. Table 1 reports the error in meters, averaged over 50timesteps (ADE) and at specific horizons (FDE). DRF-N ET significantly outperforms all baselines.6 Prediction horizon (sec) N e g a t i ve l og li ke li hood Density NetMDN (8)MDN (8)MDN (4)ConvLSTMDRF-NetDRFDensity Net 0 2 4 6 8 10
Prediction horizon (sec) M ea n m od ec oun t DRFConvLSTMDensity NetMDN (4)MDN (8) 0 2 4 6 8 10
Prediction horizon (sec) E n t r op y ( b i t s ) Density NetMDN (8)DRF 0 2 4 6 8 10
Prediction horizon (sec) E n t r op y p e r m od e ( b i t s ) ConvLSTMMDN (4)Density Net MDN (8)DRF (a) NLL (b) Modality (c) Entropy (d) Entropy per mode
Figure 6: Test metrics. DRF-N ET has low NLL (a) and captures the multimodality inherent in long-range futures (b). Discrete state space (DRF, ConvLSTM) yields the lowest NLL and entropy (c),and entropy per mode saturates. However, EPM increases with horizon for continuous MDNs (d). Ground truth data (NLL) Real detection data (NLL)Ablative model variant Mean @ 1 s @ 3 s @ 10 s Mean @ 1 s @ 3 s @ 10 sIndependent, categorical (Fully conv) 2.45 0.80 1.83 3.89 3.06 1.49 2.46 4.45+ Sequential refinement (DRR) 2.40 0.80 1.78
Table 3: Ablation study of multiple probabilistic prediction heads. Metric is NLL as in Table 1.
Model calibration
To understand overconfidence of predictive models, we compute calibrationcurves and expected calibration error (ECE) on the ground truth test set according to Guo et al. [31] bytreating models as multi-way classifiers over space. ECE measures miscalibration by approximatingthe expected difference between model confidence and accuracy. DRF has the lowest calibrationerror, with accuracy closest to the model confidence on average, as shown in Fig. 5. While somewhatoverconfident, these models could be recalibrated with isotonic regression or temperature scaling.
Multimodality and Entropy Analysis
We propose a ModePool operator to estimate the number ofmodes of a discrete spatial distribution. ModePool approximates the number of local maxima in adiscrete distribution p as follows, where the max is taken over | δ r | , | δ c | ≤ (cid:98) k (cid:99) , i.e . k × k windows:ModePool k,(cid:15) ( p ) = (cid:80) i,j p i,j =max p i + δr,j + δc p i,j ≥ (cid:15) (7)Only local maxima with mass exceeding a threshold (cid:15) are counted. ModePool is efficiently imple-mented on GPU by adapting the MaxPool filter commonly used in CNNs for downsampling. InFigure 6-b, modality is estimated with k = 5 , (cid:15) = 0 . . Given our output resolution, at most one modeper . × . m area can be counted. While the baseline MDN-4 predicts multiple Gaussian distri-butions, we observe strong mode-collapse. In contrast, DRF produces predictive posteriors that haveincreasingly multimodal predictions over horizons. Though an MDN of 8 mixtures captures somemultimodality as well, the mean number of modes is highly inconsistent over time (6-b, middle).Fig. 6-c shows the mean entropy of the predicted distributions. Entropy for DRF-N ET is the lowest.As DRF-N ET also achieves lower NLL at all future horizons (6-a), DRF-N ET predictions can beinterpreted as low bias and low variance. We combine entropy and modality into a single metric inFig. 6-d. For the discrete heads (DRF, ConvLSTM), the entropy per mode saturates. These modelscapture inherent future uncertainty by adding distributional modes e.g. high level actions rather thanincreasing per-mode entropy. This is not the case for baselines, where entropy per mode grows overtime. Qualitatively, in Fig. 7, DRF-N ET predictions remain the most concentrated over long horizons. Semantic mass ratio
Our semantic map can partition the world into three disjoint high-level classes, C = { Crosswalk , Road , Off-Road } . To interpret how well models understand the map, we measure confidence-weighted semantic accuracy , the mean predicted mass that falls on the correct map class.We also measure safety-sensitive recall , the mean mass that falls into a drivable region when the PoIis in a drivable region—performance when a PoI is on-road is very important to a self-driving car.Let c ( x ) ∈ C be the class of location x , determined by the map, and c ∗ t be the ground truth class ofthe PoI position at time t . Then, we compute metrics as follows, reported in Table 1:Accuracy ( c ∗≥ , x ≤ , Ω) = 1 T f (cid:80) T f t =1 P ( c ( x t ) = c ∗ t ) (8)Recall ( c ∗≥ , x ≤ , Ω) = 1 | SS | (cid:80) t ∈ SS P ( c ( x t ) ∈ { CW, ROAD } ) , (9)7 s5 s10 s Density NetConvLSTMDRF-Net (proposed)
MDN-8 (a) Pending jaywalk (d) Crosswalk approach(b) Ongoing jaywalk (e) Failure cases(c) Sidewalk traversal
Figure 7: Pedestrian predictions: ground truth past trajectory is green, future is black, opacity showsdensity, and color shows time horizon. MDN-4 predictions are omitted due to similarity to MDN-8;both are largely unimodal. More results in the supplementary video.where SS = { t : c ∗ t ∈ { CW, ROAD }} , the safety-sensitive timesteps. DRF-N ET significantly out-performs baselines on semantic mass ratio metrics, and most accurately predicts the type of surfacethe PoI will traverse. This suggests that DRF-N ET better uses the map, and is qualitatively reflectedby low-entropy, concentrated mass within map polygons in Figure 7. Ablation Study
We conduct an ablation study that evaluates the value of discrete predictions and ourresidual flow formulation. We study two variants of the DRF prediction head, a fully convolutionaland a discrete residual refinement (DRR) head. MDNs predict continuous mixtures of Gaussiansassuming conditional independence of future states, which can be discretized for cost-based planning.We can instead directly predict independent discrete distributions. The fully convolutional predictorprojects the spatial feature F (Section 3) into a 50-channel space representing per-timestep logitswith a 1x1 convolution on scene features. Spatial softmax produces valid distributions over thediscrete spatial support. The DRR head takes as input the discrete probability distributions output byour fully convolutional predictor and sequentially predicts per-timestep residuals in log-space withper-timestep weights. DRR thereby refines independent predictions sequentially.Table 3 shows that state space discretization and categorical prediction (fully convolutional head)has significantly better NLL than the best continuous mixture model in Table 1, a . reduction inNLL. Sequential refinement of independent predictions using DRR improves performance. However,predicting flow in the log probability space with DRF achieves the best likelihood. Qualitative Results
Figure 7-a shows predictions for a pedestrian in a challenging pre-crossingscenario. Predictive posteriors modeled by DRF-N ET (4th row) express high multimodality and con-centrated mass, with three visible high-level actions: stopping, crossing straight, or crossing whileskirting around a car. DRF-N ET also exhibits strong map interactions, avoiding parked vehicles.However, MDNs predict highly entropic, unimodal distributions, and the ConvLSTM places substan-tial spurious mass on parked vehicles. Across other test scenes, we observe that DRF-N ET constructslow-entropy yet multimodal predictions with similarly strong map and actor interactions. In Figure 7-d, DRF-N ET is the only model to correctly predict a crosswalk approach. Still, in failure cases,all models predict crossings too early, possibly due to unknown traffic light state. This could leadto more conservative self-driving vehicle plans if the pedestrians were nearby. Nonetheless, thesepedestrians and lights are distant. In this paper, we develop a probabilistic modeling technique applied to pedestrian behavior prediction,called Discrete Residual Flow. We encode multi-actor behaviors into a bird’s eye view rasterizationaligned with a detailed semantic map. Based on deep convolutional neural networks, a probabilisticmodel is designed to sequentially update marginal distributions over future actor states from therasterization. We empirically verify the effectiveness of our model on a large scale, real-world urbandataset. Extensive experiments show that our model outperforms several strong baselines, expressinghigh likelihoods, low error, low entropy and high multimodality. The strong performance of DRF-N ET ’s discrete predictions is very promising for cost-based and constrained robotic planning. Acknowledgments
We would like to thank Abbas Sadat for useful discussions during the development of this research.8 eferences [1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social lstm:Human trajectory prediction in crowded spaces. In
Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition , pages 961–971, 2016.[2] S. Becker, R. Hug, W. H¨ubner, and M. Arens. An Evaluation of Trajectory Prediction Ap-proaches and Notes on the TrajNet Benchmark.
CoRR , abs/1805.07663, 2018.[3] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social GAN: Socially AcceptableTrajectories with Generative Adversarial Networks. In
IEEE Conference on Computer Visionand Pattern Recognition (CVPR) , 2018.[4] W. Luo, B. Yang, and R. Urtasun. Fast and Furious: Real Time End-to-End 3D Detection,Tracking and Motion Forecasting With a Single Convolutional Net. In
IEEE Conference onComputer Vision and Pattern Recognition (CVPR) , 2018.[5] S. Casas, W. Luo, and R. Urtasun. IntentNet: Learning to Predict Intention from Raw SensorData. In
Conference on Robotic Learning (CoRL) , 2018.[6] M. Bansal, A. Krizhevsky, and A. S. Ogale. Chauffeurnet: Learning to drive by imitating thebest and synthesizing the worst.
CoRR , abs/1812.03079, 2018.[7] C. M. Bishop. Mixture density networks. Technical report, Citeseer, 1994.[8] H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, andN. Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutionalnetworks. arXiv preprint arXiv:1809.10732 , 2018.[9] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese. Sophie:An attentive gan for predicting paths compliant to social and physical constraints. In
Proceed-ings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1349–1358,2019.[10] D. Ridel, E. Rehder, M. Lauer, C. Stiller, and D. Wolf. A literature review on the predictionof pedestrian behavior in urban scenarios. In
Proceedings of the International Conference onIntelligent Transportation Systems , November 2018.[11] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras. Humanmotion trajectory prediction: A survey.
ArXiv , abs/1905.06113, 2019.[12] W.-C. Ma, D.-A. Huang, N. Lee, and K. M. Kitani. Forecasting interactive dynamics of pedes-trians with fictitious play. In
Computer Vision and Pattern Recognition (CVPR), 2017 IEEEConference on , pages 4636–4644. IEEE, 2017.[13] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K.Dey, and S. Srinivasa. Planning-based prediction for pedestrians. In
Intelligent Robots andSystems, 2009. IROS 2009. IEEE/RSJ International Conference on , pages 3931–3936. IEEE,2009.[14] J. Wu, J. Ruenz, and M. Althoff. Probabilistic map-based pedestrian motion prediction takingtraffic participants into consideration. In , pages1285–1292. IEEE, 2018.[15] E. Rehder, F. Wirth, M. Lauer, and C. Stiller. Pedestrian prediction by planning using deepneural networks. In ,pages 1–5. IEEE, 2018.[16] E. Rehder and H. Kloeden. Goal-directed pedestrian prediction. In
Proceedings of the IEEEInternational Conference on Computer Vision Workshops , pages 50–58, 2015. URL .[17] J. F. Fisac, A. Bajcsy, S. L. Herbert, D. Fridovich-Keil, S. Wang, C. J. Tomlin, and A. D. Dragan.Probabilistically Safe Robot Planning with Confidence-Based Human Predictions. In
Robotics:Science and Systems (RSS) , 2018. 918] A. Bajcsy, S. L. Herbert, D. Fridovich-Keil, J. F. Fisac, S. Deglurkar, A. D. Dragan, and C. J.Tomlin. A scalable framework for real-time multi-robot, multi-human collision avoidance.
CoRR , abs/1811.05929, 2018.[19] B. Yang, M. Liang, and R. Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In
Conference on Robot Learning , pages 146–155, 2018.[20] N. Djuric, V. Radosavljevic, H. Cui, T. Nguyen, F.-C. Chou, T.-H. Lin, and J. Schneider. Motionprediction of traffic actors for autonomous driving using deep convolutional networks. arXivpreprint arXiv:1808.05819 , 2018.[21] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo. Convolutional lstmnetwork: A machine learning approach for precipitation nowcasting. In
Advances in neuralinformation processing systems , pages 802–810, 2015.[22] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normal-ization. In
Computer Vision (ICCV), 2017 IEEE International Conference on , pages 1510–1519.IEEE, 2017.[23] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In
InternationalConference on Machine Learning , pages 1530–1538, 2015.[24] M. Liang, B. Yang, S. Wang, and R. Urtasun. Deep continuous fusion for multi-sensor 3dobject detection. In
Proceedings of the European Conference on Computer Vision (ECCV) ,pages 641–656, 2018.[25] E. A. Wan and R. Van Der Merwe. The unscented kalman filter for nonlinear estimation.In
Adaptive Systems for Signal Processing, Communications, and Control Symposium 2000.AS-SPCC. The IEEE 2000 , pages 153–158. Ieee, 2000.[26] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In
Pro-ceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778,2016.[27] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramidnetworks for object detection. In
Proceedings of the IEEE Conference on Computer Vision andPattern Recognition , pages 2117–2125, 2017.[28] K. Saleh, M. Hossny, and S. Nahavandi. Long-term recurrent predictive model for intentprediction of pedestrians via inverse reinforcement learning. In , pages 1–8. IEEE, 2018.[29] A. Zyner, S. Worrall, and E. Nebot. Naturalistic driver intention and path prediction usingrecurrent neural networks. arXiv preprint arXiv:1807.09995 , 2018.[30] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguingfailing of convolutional neural networks and the coordconv solution. In
Advances in NeuralInformation Processing Systems , pages 9628–9639, 2018.[31] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural net-works. In
Proceedings of the 34th International Conference on Machine Learning - Volume 70 ,ICML’17, pages 1321–1330. JMLR.org, 2017. URL http://dl.acm.org/citation.cfm?id=3305381.3305518 .[32] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.
International Conferenceon Learning Representations , abs/1412.6980, 2015.10
Appendix
In this appendix, we provide additional implementation (Section 6.1-6.2), training (Section 6.3) andevaluation (Section 6.4) details for our proposed DRF-N ET and baseline architectures. We alsoprovide a derivation of the DRF update equation (Section 6.5). In Section 3 of the paper, we described a deep convolutional neural network architecture that repre-sents our spatio-temporal scene rasterization Ω as a global feature F . This CNN architecture formsthe initial layers of the proposed model and baselines, though each network is trained end-to-end(backbone parameters are not shared across models). The backbone architecture is detailed in Fig-ure 8, below. The proposed DRF-N ET further projects the N × × × feature F into a channel space with a learned × convolutional filter for memory efficiency. + Upsample x2N ✕ ✕ ✕ Scene rasterization
Subset of timesteps shown N ✕ ✕ ✕ N ✕ ✕ ✕ N ✕ ✕ ✕ N ✕ ✕ ✕ + + Upsample x2Modified ResNet-18 with Group Normalization
Upsample-aggregate stepsto form feature pyramid ✕ ✕ ✕ Figure 8: Backbone feature pyramid network (FPN). N denotes the batch size, e.g . the number ofpedestrians of interest for inference or number of scenarios per batch for training. The input bird’s eye view (BEV) region is rotated for a fixed pedestrianof interest heading at the current time and spans meters perpendicularly and meters longitudi-nally, ahead and behind the last observed pose of the pedestrian. We set the input resolutionto . meters per pixel and the output resolution of our spatial distribution to . meters per pixel.At the input resolution, our BEV rasterization channels are each px by px. Encoding observed actor behavior
We use the object detector proposed in Liang et al. [24], whichexploits LiDAR point clouds as well as cameras in an end-to-end fashion in order to obtain reliablebounding boxes of dynamic agents. Further, we associate the object detections using a matchingalgorithm and refine the trajectories using an Unscented Kalman Filter [25]. These detections arerasterized for T p = 30 past timesteps, with ms elapsing between timesteps. At any past time t ,DRF-N ET renders a binary image D t for pedestrian occupancy where pixel D t,i,j = 1 when pixel i, j lies within a convex, bounding octagon of a pedestrian’s centroid. Other cells are encoded as .Bounding polygons of vehicles, bicycles, buses and other non-pedestrian actors are also rendered ina binary image V t . In Figure 3-c and Figure 8, we show how temporal information is encoded in thechannel dimension of tensors D and V .To discriminate the pedestrian of interest (PoI) from other actors, a grayscale image R masks thetracklet of the pedestrian to be predicted. As a convention, let the current timestep be t = 0 . If a pixel i, j is contained within the bounding polygon of the PoI at timestep t ≤ , then R i,j = 1 + γt , γ ∈ (0 , T − p ) . By doing so, the whole PoI tracklet is encoded in a single channel with decaying intensity11or older detections. This encoding allows for variable track lengths. All rasterization channels arerotated for fixed PoI orientation at t = 0 . We compute orientation with the difference of the last twoobserved locations.To allow the network to localize objects in the rasterization, two additional positional encodingchannels encode x and y coordinates as real values from − to , with value at the last known PoIlocation. Similar channels are used in [30]. Encoding semantic map
To represent the scene context of the pedestrian, DRF-N ET renders mappolygons into semantic map channels, collectively denoted as M , where each channel corre-sponds to a finely differentiated urban surface label. Crosswalks and drivable surfaces (roadwaysand intersections) are rasterized into separate channels. While sidewalks are not explicitly encoded,non-drivable surfaces are implied by the road map. Three channels indicate traffic light state, classi-fied from the on-vehicle camera with a known traffic light position: the green, red, and yellow lightchannels each fill the lanes passing through intersections controlled by the corresponding light state.Similarly, lanes leading to yield and stop signs are encoded into channels. Finally, we encode otherdetailed lanes, such as turn, bike, and bus lanes, and a combined channel for all lane markers. Indetail, the 15 channels are as follows:1. Aggregated road mask, masking all drivable surfaces2. Masked crosswalks3. Masked intersections4. Masked bus lanes5. Masked bike lanes6. All lane markers / dividers7. Masked lanes leading to stop sign8. Masked lanes leading to yield sign9. Lanes controlled by red stop light10. Lanes controlled by yellow light11. Lanes controlled by green light12. Lanes without a turn13. Right-turn lanes14. Protected left-turn lanes15. Unprotected left-turn lanesThis information is annotated in a semi-automated fashion in cities where the self-driving vehiclemay operate (Section 4.1), and only polygons and polylines are stored. For density visualization in Figure 7, and for computing discretenegative log likelihood metrics in Table 1, the MDN predicted mixture is numerically integrated bya centered approximation with sampling points for each output grid cell of size . × . squaredmeters. Discretizing the MDN allows an NLL metric to be compared between continuous predictionsand discrete predictions. Optimization
In our experiments with manually annotated trajectories, we train our models fromscratch using the Adam optimizer [32] with a learning rate of − . When using trajectories from areal perception system, we fine-tune the models learned using the ground truth data to better deal withmissing pedestrians and detector/sensor noise. Each training batch includes pedestrian trajectories.All experiments are performed with distributed training on GPUs.
To compute the number of modes (local maxima) in a distribution, we pro-posed the ModePool k , (cid:15) operator. Our proposed operator in fact overestimates modality for MDNs,12specially for the Density Network, at short timescales due to quantization error and the fixed win-dow size. To compute modality of a continuous distribution, we discretize the distribution. When thedistributions are very long and narrow, as in Density Network short term predictions, multiple modescan be registered. Despite this overestimation, models with the proposed discrete prediction space(ConvLSTM, DRF-N ET ) expressed higher multimodality than the MDNs. We derive Equation (3), the discrete residual flow update equation, as an approximation for explicitmarginalization of a joint state distribution. According to the law of total probability, p x t ( x t | Ω) = (cid:88) x t − p x t , x t − ( x t , x t − | Ω) (10) = (cid:88) x t − p x t | x t − ( x t | x t − , Ω) p x t − ( x t − | Ω) (11)Equation (11) can be seen as a recursive update to the previous timestep’s state marginal. Recall that x t is a categorical random variable over K bins. Instead of representing the pairwise conditionaldistribution p ( x t | x t − , Ω) and conducting the summation once per output bin at O ( K ) cost pertimestep, we approximate (11) with a pointwise update, p x t ( x t | Ω) = (cid:88) x t − p x t | x t − ( x t | x t − , Ω) p x t − ( x t − | Ω) (11) = (cid:88) x t − p x t | x t − ( x t | x t − , Ω) p x t − ( x t − | Ω) p x t − ( x t | Ω) p x t − ( x t | Ω) (12) ≈ Z t ψ t ; θ t (cid:0) x t , p x t − ( · | Ω) , Ω (cid:1)(cid:124) (cid:123)(cid:122) (cid:125) Exponentiated residual p x t − ( x t | Ω) (13)where Z t is a normalization constant, and ψ t ; θ t is a parametric approximator for the summation thatwe refer to as the residual predictor . In principle, a sufficiently expressive residual predictor canmodel the summation exactly. While the residual is applied as a scaling factor in Equation (13),the residual becomes more natural to understand when the recursive definition is expressed in logdomain, completing the derivation, log p x t ( x t | Ω) = log p x t − ( x t | Ω) + log ψ t ; θ t (cid:0) x t , p x t − ( · | Ω) , Ω (cid:1) − log Z t (14)We construct log ψ t ; θ t such that it can be computed in parallel across all locations x t , and suchthat the update to log p x t ( · | Ω) is an elementwise sum followed by normalization. In DRF-N ET , log ψ t ; θ t is instantiated with a neural network that outputs a 2D image indexable at these locations(Figure 4). Then, the update (14) incurs O ( K ) cost per timestep.With this lens, the baseline fully convolutional predictor and the mixture density networks, whichassume conditional independence x t ⊥ x t − | Ω , directly approximate the marginal: log p x t ( x t | Ω) = log ψ t ; θ t ( x t , Ω) − log Z t (15)The baseline ConvLSTM propagates a cell and hidden state between steps and shares parameters ofthe predictor, without sampling from intermediate marginals: log p x t ( x t | Ω) = log f φ ( h t ) − log Z t (16) h t , c t = ψ θ (cid:0) p x t − ( · | Ω) , h t − , c t − (cid:1) h = F (Ω) , c is a parameterDiscrete residual flow retains most of the benefits of the independence assumption, i.ei.e