Satellite imagery analysis for operational damage assessment in Emergency situations
Alexey Trekin, German Novikov, Georgy Potapov, Vladimir Ignatiev, Evgeny Burnaev
SSatellite imagery analysis for operationaldamage assessment in Emergency situations
Alexey Trekin , German Novikov , Georgy Potapov , Vladimir Ignatiev , andEvgeny Burnaev Skolkovo institute of Science and technology, Moscow, Nobel st. 1, Russia, [email protected] ,WWW home page: http://crei.skoltech.ru/cdise/aeronet-lab/
Abstract.
When major disaster occurs the questions are raised how toestimate the damage in time to support the decision making process andrelief efforts by local authorities or humanitarian teams. In this paper weconsider the use of Machine Learning and Computer Vision on remotesensing imagery to improve time efficiency of assessment of damagedbuildings in disaster affected area. We propose a general workflow thatcan be useful in various disaster management applications, and demon-strate the use of the proposed workflow for the assessment of the damagecaused by the wildfires in California in 2017.
Key words: remote sensing, damage assessment, satellite imagery, deeplearning, emergency response, emergency mapping
In emergency situations like devastating wildfires, floods, earthquakes or tsunami,decision makers need to get information about possible damages in residentialarea and infrastructure most rapidly after the event. One of the most valuablesources to get such information from, is the Earth Observation systems, whichinclude satellite and aerial remote sensing, since it can be captured shortly af-ter the disaster without all the risks related to the ground observations. Thecombination of this information with statistical and ground observation datacontributes to even a better valuation of physical and human losses caused bydisaster [2].There are several international programs that are arranged to support theinformation exchange during and after disasters such as UNOSAT (UNITAR Op-erational Satellite Applications Program) [4], Space disaster charter [1], Human-itarian Openstreetmap Team (HOT) [3] or Tomnod [5] which is the Digitalglobesatellite company crowdsourcing platform. All these are useful initiatives pro-viding tools and activation workflows for emergency mapping done by specialistsor by volunteers in a collaborative way [8, 9].This method of mapping of the imagery could be time-consuming since itrequires some qualification and takes time to digitize all the damages manually,particularly if the affected area is quite large and the objects are relatively small a r X i v : . [ c s . C V ] F e b Alexey Trekin et al. and scattered as it is for private houses in the residential area. For example,Californian wildfires past year caused significant damages in Ventura and SantaBarbara counties. The fires destroyed at least 1,063 and 5,643 structures in theseareas respectively [11]. The significant delay in time of Emergency Mappingalso might be caused by the availability of remote sensing data which has it’sphysical limitations (cloudiness, day time, resolution etc.) as well as commercialones (terms of use, costs etc.) Needless to say that in the post-disaster recoverystrategy the time is the key factor. Thats why we consider apply the MachineLearning and Computer Vision approach to the processing of Satellite and Aerialimagery to detect main damages and reduce the time costs.
Among the existing solutions for Emergency Mapping of disaster areas its worthto mention HOT that allows mapping of the chosen area in a collaborativeway. Taking data for different areas, where HOT campaigns were activated, weestimated that the mapping process even in the areas of emergency that attracta lot of public attention, like Nepal earthquakes, takes several months to achievethe whole coverage (see figure 1).
Fig. 1.
Time distribution of the building features created by OSM users for Kathmanduregion (source - osm-analytics.org)
Following the news of this incident we found several related media publica-tions that provide assessments of damages. We assume that the work was doingmanually on satellite images - comparing to the date of incident (Dec. 4) it’staken about six days to prepare maps for the article (Dec. 11 LA Times articleupdate) [12]. That most probably is caused by the amount of work needed tofind appropriate data sources and make a damage map. amage assessment 3
UNOSAT Rapid mapping service [4] is a framework of United Nations Insti-tute for Training and Research in the field of emergency mapping. Even thoughits quite challenging to estimate the time efficiency by their results since thereis no clear definition of what states for “rapid”. Usually the temporary lag ofUNOSAT maps is two days from the satellite imagery acquisition date.One of the keys to the solution of the problem might be a deep learningapproach. In the last few years the deep convolutional networks became a ro-bust solution for many problems concerning image analysis. Different variantsof the networks are able to solve the problems of semantic segmentation, objectclassification and detection [14, 17, 18, 19].The main drawback of this class of methods is that the deep convolutionalnetworks need a big amount of training (previously manually marked groundtruth) data. On the one hand, we can pre-train the method using the dataabout the other event of the kind, that took place in the past. But the resultsof this kind of training may be unpredictable due to the difference between thedata in the training and test cases. These issues are concerned in our workflowthat is proposed in the following section.
The main problem we want to deal with is to decrease the time needed to retrievecrucial information for decision making in emergency situations when the properremote sensing data is available. We propose the following workflow:1. Determine the case of interest. The deep learning methods work significantlybetter when the objects of interest are similar to each other, so the caseshould be narrow, for example burned houses or flooded roads.2. Create a training dataset. The deep learning methods need a training dataso they could learn the objects of interest and their properties. The trainingdataset consist of the real data (in our case, two aerospace images, one takenbefore the catastrophic event, and the other - after the event) and the labelsthat annotate and outline every damaged structure of the type.3. Train and validate a deep learning method using the dataset. The method(or model) extracts the information from the training dataset. Its abilityto detect the damaged objects of interest is validated using a subset of thetraining data. This pre-trained model will be used in every case of the forth-coming emergency of the given type.4. Obtain information of a new emergency case. This is where our methodstarts working. The data should be of the same or similar type (spatialresolution, target objects, color depth) as that used for training, this is acritical requirement for the model to work properly.5. Fine-tune the model for the new case. Despite the similarity of the data, themodel may be unable to process them correctly due to small differences, forexample different sunlight. The fine-tuning can be done using automaticallyannotated data from the new case, or using the manual markup for a smallsubset of the data.
Alexey Trekin et al.
6. Run the automatic image processing. Now that the model is trained for thecase, we can make an automated processing of the rest of the data and havethem ready for the decision making.Using this approach, we need to spend some time for creation of the referencetraining dataset, but normally it should be made before the emergency event.Then, after the event, the amount of work needed is much less that allows us topropose a fast working and thus efficient solution.
To validate and demonstrate the workflow, we have chosen the case of wildfiresin two areas of Ventura and Santa Rosa counties, California, USA, where manyhouses were burned to the ground in 2017. The advantage of this choice is justi-fied by the availability of hi-resolution data provided by Digitalglobe within theirOpen Data Program [10]. In the following section we will follow our workflow onthe case, and describe both general principles of the deep learning applicationto the imagery and our steps in the particular case.
In our research the case of interest is houses destroyed by fire. A typical exampleof the object of interest is depicted in figure 4.1. (a) Before (b) After
Fig. 2.
Satellite images of buildings before and after the fire eventamage assessment 5
It is worth noting, that the case should be restricted as narrow as it is possiblefor it makes a big difference when speaking about the deep learning methods.For example, if we train the method on the images like this, where the houses arecompletely destroyed, it will not be able to detect partially damaged buildings.Also the type of building and even the rooftop material can change the resultsignificantly.
The training area is chosen in the Ventura, Santa Barbara, California, that wasseverely affected by the Thomas Fire in the December, 2017 (see figure 4.2). (a) map (b) markup
Fig. 3.
Training area in Ventura and the resulting markup (Openstreetmap, Stamendesign; Digitalglobe. Bounding box coordinates: -119.2278 34.2836 -119.1916 34.3065)
The preferable source for high-resolution data is the Digitalglobe Open DataProgram. This program is one of the few sources of free high-resolution data, andthe data is distributed early after the event [10]. However, in the case of SouthCalifornia the existing Openstreetmap (OSM) mapping which was used as theinitial input for the markup is based on Google Maps / Bing Maps imagery thatis better orthorectified, so that the image coordinates differ, as it can be seen inthe figure 4.2. This makes existing map not as good source of the ground truth.Due to these reasons, we had to use the Google Maps imagery, that is similarto the Digitalglobe open data in terms of image characteristics (similar spatialresolution, also 3-band 8-bit per band images), but both pre-event and post-eventimages are available, and the better orthorectification leads to good alignmentwith the OSM.The crowdsourced mapping data from OSM were not full and did not containthe information about burned buildings, so it was necessary to improve the OSMmarkup before using it as the ground truth for training the deep convolutionalnetwork. We facilitated the manual work by using of OSM as the source of initial
Alexey Trekin et al.
Fig. 4.
Misalignment of the Digitalglobe image with OSM markup vector features, selecting all the ones tagged as building. All the extracted fea-tures than were checked through the visual inspection and annotated with theappropriate additional tag “disaster”=“damaged area” if the one was destroyedby the fire. To complete training dataset we used cartographic tools as Open-streetmap ID which is open source editor for online mapping for OSM [6]. Thefinal markup contains 760 not damaged buildings and 320 ruined buildings (seefigure 4.2) was exported in GeoJSON format using OSM API and additionallyprocessed using our Python stack tool to convert and rasterize vector data into1-band pixel masks.
We used a semantic segmentation approach to the change detection. The seman-tic segmentation of an image results in a mask of the pixels that are consideredto be of the target class or classes. In our case, when we have two images - beforeand after the event - we can gain maximum from the given data if we stack themtogether and make a 6-band image (3 bands before and 3 bands after). A convo-lutional network for change detection was built in the encoder-decoder manner,which has great success in solving semantic segmentation problems [14]. For amodel that works with pairs of 3-band images, one could use a single 6-channelencoder, but this would not allow the use of a transfer-learning technique tospeed up learning and improve the final quality of the results, so the model wasbuilt on a two-stream encoder, each of which looked at its own 3-band image andone common decoder. This approach made it possible to use the pre-trained on“ImageNet” classification dataset [13] weights for the RGB images independentlyin each of the branches of the encoder.Validation on the part of the Ventura dataset that was not used for traininggave appropriate results, see figure 4.3. Pixel-wise F .
859 and for the class of unburned buildings is 0 . amage assessment 7 Fig. 5.
Results of the change detection on the validation subset of data in Ventura.Left: image taken before fires, center: image taken after fire, right: segmentation resultsblack - non-damaged, gray - damaged buildings
We consider the fire in Santa Rosa, California (Tubbs Fire, October 2017) as the“new case” of the same type of events (see figure 4.4). The Open Data programhas images both before and after the fire event, so we can use them for the test.
Fig. 6.
A map of the new zone of the Tubbs fire in Santa Rosa, California
As the data in this case have similar characteristics, we tried the imagesegmentation with the model as is, without any changes. The result is unsatis-factory, however it does have some correlation with the real data. This can becaused by differences in season, solar angle, image preprocessing difference, or bysome difference in she structure of the residential areas themselves. For example,buildings in Santa Rosa are closer to each other. The example of the results seein 4.4
Alexey Trekin et al.
Fig. 7.
Results of the change detection on the test subset of data in Santa Rosawithout fine-tuning. Left: image taken before fires, center: image taken after fire, right:segmentation results black - non-damaged, gray - damaged buildings
The results above show that we need to train the model for the new area. Inorder to do this, we make a new small dataset in a part of the Santa Rosa,see figure 4.5. It contains 146 burned and 137 undamaged houses, so it requiresfar less time and effort. The preparation of the dataset took about an hour ofmanual markup by one person.
Fig. 8.
Small dataset for fine-tuning of the net to the new data. White - unburnedbuildings, black contours - burned buildingsamage assessment 9
Switching from one part of dataset to another, the results of the model weregreatly deteriorated, but the dense marking of just less than 300 houses on newimages allowed to improve the quality on the whole new data and reach almostthe same result for 10 minutes of additional training.
The rest of the Santa Rosa region of interest was processed automatically by thetrained model. The example of the result taken from the test zone in the centerof Santa Rosa town is shown in figures 4.6, 4.6. It can be clearly seen that nonfine-tuned method tends to merge the regions of the separate buildings into onearea, while after the fine-tuning the resulting regions can be easily separated atthe post-processing stage. (a) Before event (b) After event
Fig. 9.
An example of the test area image before and after the fire
After the fine-tuning, the change detection method can give very good resultson image segmentation, and even give a good chance to distinguish betweenseparate houses that is very important in the task of damage assessment whenit is necessary to estimate the accurate number of damaged structures.Note that the segmentation approach is more robust than the detection onebecause it allows to estimate the area of the target changes, that can be necessaryin other emergency cases like floods, blockages, fire damage to crops or forestsetc.
Fig. 10.
Results of the image segmentation before and after fine-tuning
The manual markup of our Ventura training area (figure 4.2 ) should take about1 . − At the current stage we have developed a workflow and a method of the damagedareas segmentation. In the further research we will continue the development ofthe segmentation method to increase its accuracy and robustness to the datacharacteristics changes.The method can be also extended to the problem of instance segmentation,that is distinguishing between separate objects, counting the objects, and con-verting them to the map that can be used online and in GIS applications. amage assessment 11
We will apply the approach to the Open Data in the case of new events ofthis domain, the other types of disasters such as floods and tsunami, and willextend the training dataset to extrapolate this approach to the other cases andterritories.
Weve formulated the problem based on the research of the tools and frameworksfor disaster mapping. Based on the problem, we proposed a workflow involvingdeep learning and use of open data for the emergency mapping.Weve created the training and test datasets for California fires, which meansthe raster mask of the vector features of damaged and non damaged buildings inthe area and the appropriate pre- and post-event imagery to develop a changedetection method and validate the approach.We developed a method of change detection, based on convolutional neuralnetworks, that is able to make semantic segmentation of the area subjected tomassive fires, mapping burned buildings. The method can be easily extended tothe new areas and data sources with a little training for the data peculiarities(fine tuning).The workflow turned up to give substantial profit in terms of time neededfor emergency mapping and in the future we will extend it to the other cases.