Ocean Eddy Identification and Tracking using Neural Networks
Katharina Franz, Ribana Roscher, Andres Milioto, Susanne Wenzel, Jürgen Kusche
OOCEAN EDDY IDENTIFICATION AND TRACKING USING NEURAL NETWORKS
Katharina Franz, Ribana Roscher, Andres Milioto, Susanne Wenzel, J¨urgen Kusche
University of Bonn, Institute of Geodesy and Geoinformation, Nussallee 15+17, 53115 Bonn, Germany
ABSTRACT
Global climate change plays an essential role in our daily life.Mesoscale ocean eddies have a significant impact on globalwarming, since they affect the ocean dynamics, the energy aswell as the mass transports of ocean circulation. From satel-lite altimetry we can derive high-resolution, global maps con-taining ocean signals with dominating coherent eddy struc-tures. The aim of this study is the development and evalu-ation of a deep-learning based approach for the analysis ofeddies. In detail, we develop an eddy identification and track-ing framework with two different approaches that are mainlybased on feature learning with convolutional neural networks.Furthermore, state-of-the-art image processing tools and ob-ject tracking methods are used to support the eddy tracking. Incontrast to previous methods, our framework is able to learna representation of the data in which eddies can be detectedand tracked in more objective and robust way. We show thedetection and tracking results on sea level anomalies (SLA)data from the area of Australia and the East Australia current,and compare our two eddy detection and tracking approachesto identify the most robust and objective method.
Index Terms — Mesoscale eddies, semantic segmenta-tion, convolutional neural networks, optical flow
1. INTRODUCTION
The global ocean circulation is substantially affected by char-acteristic vortices, called mesoscale ocean eddies, moving cir-cularly in parts of the Earth’s great currents. Their main con-tribution to ocean circulation is due to the transport of a hugeamount of kinetic energy. Hence, their detection and track-ing is highly relevant in terms of oceanography and climatechange studies, respectively. Pattern recognition in satelliteimageries is a common task in remote sensing, where deeplearning methods make the large amounts of data collectedfrom space considerably more usable than traditional machinelearning algorithms. These approaches allow the recognitionof objects and coherent patterns from data, wherefore we de-velop a convolutional neural network (CNN)-based eddy de-tection and tracking framework.Mesoscale eddies are small circular currents that showcharacteristic cyclonal and anticyclonal patterns in globalmaps, e.g. sea level heights [1], and their automatic detectionand identification is an active research topic. Several methods have been developed to approach the formulated detectiontask, some of which are geometrically inspired [2], or on thebasis of signal decompositions [3]. In particular, the Okubo-Weiss method [4, 5] is the most popular physical procedureto approach this problem, due to its physical interpretabil-ity, but open questions remain with respect to the thresholddefiniton [6]. On the other hand, deep learning algorithmsare a breakthrough innovation to approach computer visiontasks such as object recognition, and have recently been usedfor oceanographic applications ([7]). However, neural net-works require a large amount of reference data, usually onlyavailable at a high cost, in order to learn an adequate modelfor recognition. Therefore, these supervised approaches arecombined with data generation or data augmentation proce-dures to overcome this problem. [8, 9], for example, proposeautomated techniques to generate training data. In contrast[10] suggest transfer learning approaches based on pretrainednetworks on large openly available datasets that are fine-tunedfor the current classification task which significantly reducesthe amount of training data needed. [11] develop a frame-work for the classification of multispectral images using deeplearning techniques by segmentation with CNNs, and con-sider the scarcity of annotated satellite data by proposing thesynthesization of reference data as an automated procedure.In this paper, we propose an eddy detection and track-ing framework combining feature learning by CNNs with anestablished image processing tool, the Kanade-Lucas-Tomasi(KLT) [12] feature tracker. Furthermore, we compare thiswith a recurrent neural network (RNN), in particular a long-short term memory (LSTM), trained for eddy identification.We utilize the Okubo-Weiss method to tackle the lack of an-notated data by detecting a few, yet precise eddies, which areused as training data. By combining these approaches, we areable to achieve a high recall as well as high precision, whichtradeoff is generally a problem for ordinary eddy detectionmethod such as the Okubo-Weiss procedure.
2. DATA
Our study site covers seas around Australia and the SouthEquatorial current. The dataset contains high-resolution,global sea level anomaly maps (SLA), which are prepro-cessed products from the ESA Climate Change Initiative(ESACCI) [13]. The gridded SLA maps contain the mesoscalecoherent eddy features and variabilities and enable the dis- a r X i v : . [ c s . C V ] M a y ig. 1 . Workflow:
Annotated data and one-channel SLAimages are generated serving as input to the detection mod-ule and the tracking module. Two approaches, a CNN+KLT-Tracker and a convolutional LSTM are applied and evaluated.tinction between cyclonal and anticyclonal eddies. The dataset covers daily data from January 1993 until December 2014,merged from several altimetry missions. The training and val-idation data sets are generated automatically (cf. Subsec. 3.1),and state independent regional patches from the global SLAmaps with a length of 365 days per year. The validation dataset is a smaller subset of the training set. We represent thedata as 1-band images with a spatial resolution of . ◦ in a × and in a × grid.
3. METHOD
Our framework combines CNNs for feature learning with twodifferent tracking approaches, namely a tracking realized onthe basis of an optical flow estimation with KLT-Tracker, andan RNN. It contains different modules that are visualized in aflowchart in Figure 1.1.
Data generation:
The pseudo-reference data modulegenerates annotated data, which we split into training,validation and test sets. Corrected eddy detections fromOkubo-Weiss method serve as basis for this module, seeSec. 3.1.2.
Detection:
The CNN-module learns a model from whichwe can detect eddy cores at a single epoch, see Sec. 3.2.3.
Tracking:
The KLT-tracker tracks the detected eddies us-ing a sparse optical flow. On the other hand, a RNN usesconvolutional LSTM units to track the eddy cores, see 3.3.4.
Evaluation:
Since no reference data is given, we evalu-ated our results qualitatively with focus on plausability.
In order to tune and train the neural network, we require an-notated reference data, which generally does not exist in ourapplication. Consequently, we propose an automated label-ing process to generate precise pseudo-reference data basedon the results of the Okubo-Weiss method [4, 5, 14].
Fig. 2 . CNN architecture:
Blue: Conv, BatchNorm andReLU-activation layers; Green: pooling layers; Orange: up-sampling layers; Red: softmax-activation layer.The Okubo-Weiss method is an established algorithm foreddy detection. It aims at the computation of a physically mo-tivated parameter that represents the balance between vortic-ity scales and shear strain rates. The value of Okubo-Weiss-parameter decides whether the vorticity dominates over strain,where an eddy core is defined as vorticity dominated area.Therefore, the choice of the threshold on the Okubo-Weiss-parameter defines the precision and recall of the eddy identi-fication. Based on this, we define a threshold which providesa few, yet precise eddy cores, which are corrected manuallyand used as training and validation data for the CNN. Wetreat the task as a two-class problem such that each cell of thegrid-based data is assigned to the class ’eddy’ or ’non-eddy’,nevertheless, the approach can be extended to distinguish be-tween anticyclonic, cyclonic and non-eddies. Since each eddycore can only be defined with a certain precision, we definea circular domain around each detection and treat it as eddycore rather than defining only one grid cell.
Figure 2 visualizes the architecture of our CNN, which we re-alize as encoder-decoder. We use five CNN building blockswith convolutional, batch normalization and activation layers,and a pooling layer, respectively, in the encoder part. The de-coder part is defined in a similar way as the encoder, exceptfor the downsampling layer, for which we use an upsamplinglayer at the beginning of each building block. Our input aregridded one-band image subsets. The output of the CNN arethe probabilities of the classes ’eddy’ or ’non-eddy’, wherethe output has the same dimension as the input image due tothe chosen encoder-decoder architecture. In order to localizeeddy cores, we extract local maxima using non-maxima sup-pression, and define these points as eddy cores. Furthermore,the eddy core identification can be optimized and improved interms of robustness by extending the feature space, i.e., withvelocity information.
In order to track identified eddies over time, we developedtwo approaches. First, we utilize the KLT-tracker with sparseoptical flow estimations and, besides, we apply a convolu-tional LSTM network. Optical flow is defined as distribu-tion of apparent velocities and motion patterns, and there-fore leads to spatial information and the changes of spatialrrangements of objects. The KLT-tracker is a feature trackerthat traces distinctive points over time based on the motionfield of optical flow estimations [12]. Thus, the differencesin motion between two consecutive time steps are definedas I ( x, y, t ) = I ( x + ∆ x, y + ∆ y, t + ∆ t ) , where I is thegridded SLA data with position ( x, y ) at time t and the cor-responding differentials ∆ x , ∆ y and ∆ t . In our study, thepositions are estimated by the detection method. As secondapproach, we use RNNs which directly process time-varyingimage sequences of SLA data. In detail, we apply convolu-tional LSTMs, which are able to model spatiotemporal depen-dencies.
4. EXPERIMENTS AND DISCUSSION4.1. Experimental Setup
We investigate the CNN-based detections regarding robust-ness and amount, and their suitability as input into a KLT-tracker with sparse optical flow estimation. In contrast, weaddress the stand-alone tracking approach with convolutionalLSTMs. For our study, the threshold for the Okubo-Weissmethod is set to . σ W . Here, σ W is the standard devia-tion of the Okubo-Weiss parameter. The radius of the eddycores used as pseudo-reference data is defined as grid cellsin each direction which corresponds to km. For trainingthe CNN, we use a kernel size of grid cells which is equalto the stride. Furthermore, we have trained the CNN on epochs and used a batch size of grid cells. However, theconvolutional LSTM is trained on epochs since the netneeds to fit more hyperparameters. For those computations,we take advantage of a GPU’s power, although, we need to re-size the images to × grids. We extract training data fromall time steps, except March , which we use for testing.We use the data from March st Figure 3 shows the results at the East Australian and the SouthEquatorial currents for each point in time. The left frame ofthe sequence presents the detected eddy cores at March st Table 1 . CNN detections and KLT-tracking: number of de-tected and tracked eddy cores (first row), averaged distancesof eddy movement (second row)Day Day Day Day Day Eddies
100 87 71 64 62
Av. distance - km km km km From the CNN output, we extract most likely cells (see Sec.3.2). These detected eddy positions are colored in red show-ing probabilities of about
60 % and more. Overall, we detect eddies at March st . The CNN returns different intensi-ties, i.e., a weaker eddy signal with low probabilities of about
62 % is to be found in the South of Australia, despite this, weare able to identify the relevant eddy structures. The great-est concentration of eddy cores is detected at the warm EastAustralian Current. By comparison, the probabilities of
69 % are slightly higher than those at the South of Australia. Weakeddy signals indicate high variabilities. In consequence, themost eddies might vanish in those fields and they may provideno sufficient tracking results. Furthermore, the CNN detectsthe highest probabilities near Papua-New Guinea in the North,and in the Northeast of Australia near the coast. Here, we ex-tract eddy cores, which are located in the high-probabilityareas, as can be seen in Fig. 3(a). Nevertheless, each identi-fied eddy core is characterized by an uncertainty, due to thenoisy pseudo-reference data and the filtering algorithm ap-plied on the CNN probabilities. The results of the KLT-Tracker are presented in the SLA mapsof day up to day in Fig. 3. Tab. 1 counts the num-ber of tracked eddy cores. Over time the number of tracksis reduced, hence, some eddies vanish in regions with highvariability. These eddies are located at the East AustralianCurrent and the Antarctic Circumpolar Current, in particu-lar. Although, there is no apparent movement across the gridcells of the tracked eddies in Fig. 3, we can analyze the move-ment of the eddies using the subpixel estimation of the KLT-tracker. We computed the Euclidean distances between as-signed points, and report the averages in Tab. 1. A drawbackof the KLT-tracker is a potential mismatch of tracked eddiesdue to their similar pattern, since all detections are tracked in-dependently. In addition, the uncertainties of the estimationsplays a role in imprecisions along the time series, too.Fig. 4 shows our preliminary results of using a convo-lutional LSTM for eddy tracking. The results indicate thatareas with multiple eddies can be detected. Due to memoryand capacity restrictions of our used GPU, the spatial resolu-tion is lower in comparison to our previous results, makinga detection of single eddies difficult. However, we see this ig. 3 . : Eddy detections with CNN and tracking results obtained by KLT-tracker: from left to right: March st , March nd and March rd red dots : eddy cores) Fig. 4 . : Eddy detections and tracking results using a con-volutional LSTM: from left to right: March st , March nd and March rd yellow: high probabilities; blue : lowprobabilities; black : land areaapproach as a promising direction for future research, since itis capable to learn an appropriate eddy representation and thedistribution of the eddy populations in order to track intenseeddy signals.
5. CONCLUSIONS AND FUTURE WORK
The convolutional LSTM reveals plausible and robust pre-liminary results. Hence, it is preferable to develop a con-volutional LSTM that is able to distinct between single cy-clonal and anticyclonal eddies. Furthermore, in order to makeeddy detections more robust and precise, we propose to ex-tend the feature space with dense optical flow information ateach point. Further data like vorticity or sea surface temper-atures are other options provided they are independent fromSLA data. However, the spatial resolution is not adequate,such that the analysis of along-track data in place of griddedmaps will be considered in the future .
6. REFERENCES [1] D. B. Chelton, L.-L. Fu, P.-Y-Le Traon, and R. Morrow, “EddyDynamics From Satellite Altimetry,”
Oceanography , vol. 23,no. 4, pp. 14–25, 2010.[2] I. A. Sadarjoen and F. H. Post, “Detection, quantification, andtracking of vortices using streamline geometry,”
Computer &Graphics , vol. 24, pp. 331 – 341, 2000. [3] A. M. Doglioli, B. Blanke, S. Speich, and G. Lapeyre, “Track-ing coherent structures in a regional ocean model with waveletanalysis: Application to Cape Basin eddies,”
J. Geophys. Res. ,vol. 112, no. C5, 2007.[4] A. Okubo, “Horizontal dispersion of floatable particles in thevicinity of velocity singularities such as convergences,”
DeepSea Research and Oceanographic Abstracts , vol. 17, pp. 445–454, 1970.[5] J. Weiss, “The dynamics of enstrophy transfer in two-dimensional hydrodynamics,”
Physica , vol. 48, pp. 273–294,1991.[6] J. H. Faghmous, I. Fenger, Y. Yao, R. Wrmka, A. Lindell, andV. Kumar, “A daily globl mesoscale ovean eddy datset fromsatellite altimetry,”
Scientific Data , vol. 2, October 2014.[7] R. Lguensat, M. Sun, R. Fablet, E. Mason, P. Tandeo, andG. Chen, “EddyNet: A deep neural network for pixel-wiseclassification of oceanic eddies,”
CoRR , vol. abs/1711.03954,2017.[8] A. Vallet and H. Sakamoto, “A multi-label convolutional neuralnetwork for automatic image annotation,”
JIP , vol. 23, no. 6,pp. 767–775, 2015.[9] D. Pathak, R. B. Girshick, P. Doll´ar, T. Darrell, and B. Hariha-ran, “Learning features by watching objects move,”
CoRR , vol.abs/1612.06370, 2016.[10] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lip-son, “How transferable are features in deep neural networks?,”
CoRR , vol. abs/1411.1792, 2014.[11] Ronald Kemker and Christopher Kanan, “Deep neural net-works for semantic segmentation of multispectral remote sens-ing imagery,”
CoRR , vol. abs/1703.06452, 2017.[12] B. D. Lucas and T. Kanade, “An Iterative Image Registra-tion Technique with an Application to Stereo Vision,” in
Proc.DARPA Image Understanding Workshop , April 1981, pp. 121 –130.[13] ESA, “Data Standards Requirements for CCI Data Producers,”Tech. Rep., European Space Agency, 2015.[14] Y.-H. Cheng, C.-R. Ho, Q. Zheng, and N.-J. Kuo, “Statisti-cal Characteristics of Mesoscale Eddies in the North PacificDerived from Satellite Altimetry,”