Active Learning for Event Detection in Support of Disaster Analysis Applications
NNoname manuscript No. (will be inserted by the editor)
Active Learning for Event Detection in Support of Disaster AnalysisApplications
Naina Said · Kashif Ahmad · Nicola Conci · Ala Al-Fuqaha · Received: date / Accepted: date
Abstract
Disaster analysis in social media content is one ofthe interesting research domains having abundance of data.However, there is a lack of labeled data that can be usedto train machine learning models for disaster analysis appli-cations. Active learning is one of the possible solutions tosuch problem. To this aim, in this paper we propose and as-sess the efficacy of an active learning based framework fordisaster analysis using images shared on social media out-lets. Specifically, we analyze the performance of differentactive learning techniques employing several sampling anddisagreement strategies. Moreover, we collect a large-scaledataset covering images from eight common types of natu-ral disasters. The experimental results show that the use ofactive learning techniques for disaster analysis using imagesresults in a performance comparable to that obtained usinghuman annotated images, and could be used in frameworksfor disaster analysis in images without tedious job of manualannotation.
Keywords
Disasters analysis · active learning · multimediaretrieval · Uncertainty sampling · query by committee Natural disasters, such as floods and earthquakes, may causesignificant loss in terms of human lives and property. In suchsituations, an instant access to relevant information may helpwith timely recovery efforts. In recent years, social mediaoutlets have been widely utilized to gather disaster relatedinformation [3]. However, the use of social media content
University of Trento, ItalyHamad Bin Khalifa UniversityE-mail: [email protected] also comes with lots of challenges. One such challenge is fil-tering out irrelevant information. To this aim, several frame-works have been proposed in the recent literature that relyon different classification and feature extraction techniques.One of the requirements of classification applications is theavailability of sufficient training samples. However, annota-tion of training samples is a tedious and time consuming job,which requires lots of efforts.One of the possible solutions to reduce human labor indata annotation is the use of active learning techniques. Ac-tive learning has been widely utilized in a wide range of ap-plication domains having large quantities of unlabeled dataand less quantities of labeled data. Such domains includeNatural Language Processing (NLP), multimedia analysisand remote sensing [17,21,24,2,25]. Active learning tech-niques have been recently used with Convolutional NeuralNetworks (CNNs) and Long-short Term Memory (LSTM)based frameworks to improve their overall performance [21,15]. Disaster analysis is relatively a new application that stilllacks large collections of labeled data [20]. We believe itcould benefit from active learning.In this paper, we study and analyze the efficacy of utiliz-ing active learning techniques in disaster analysis in socialmedia images by employing and evaluating the performanceof different active learning techniques in terms of classifi-cation accuracy. We mainly focus on the most commonlyused scenario of active learning, namely, pool-based sam-pling that fits well in our disaster analysis task. In pool-based sampling, samples are drawn from a pool of unla-beled images into the initial small labeled training set. Un-der the above mentioned settings, we rely on two most com-monly used query techniques; namely, (i) uncertainty sam-pling and (ii) query by committee. We further evaluate theperformance of these techniques with different sampling anddisagreement strategies. For uncertainty sampling, we em-ploy three different sampling strategies; namely, least con- a r X i v : . [ c s . C V ] S e p Naina Said et al. fidence (LC), margin sampling (MS) and entropy sampling(ES). On the other hand, for query by committee based ac-tive learning approach, we explore and evaluate the capa-bilities of this approach with three different disagreementstrategies; namely, vote entropy (VE), consensus entropy(CE) and max disagreement (MD). Moreover, we analyzeand evaluate the performance of these methods using dif-ferent number of queries by including a single image in thetraining set from the unlabeled pool of images to analyzehow quickly each of the methods attains maximum accu-racy.To the best of our knowledge no prior works exploredsuch detailed analysis of active learning techniques in therelative new domain of disaster analysis applications. More-over, considering the lack of large-scale (in terms of im-ages as well as the number of disaster types/classes covered)benchmark datasets in the domain, we also provide a bench-mark dataset containing a large number of images from mostcommon types of natural disasters, as detailed in Section 6.The main contributions of this work are:(i) Stemming from the fact that machine learning techniquesare driven by training data and annotating large volumesof data is a tedious and time consuming job, we carry outan analysis and evaluation study of active learning tech-niques with diversified set of sampling/disagreement strate-gies in support of disaster analysis applications.(ii) Through the introduction of the active learning techniques,we demonstrate that comparable accuracy can be achievedwith active learning without involving human annotatorsin the tedious job of annotating large training sets, andactive learning could be used in disaster analysis frame-works to obtain better results in scenarios where less an-notated data is available.(iii) We also analyze and evaluate the performance of themethods using different numbers of queries/iterations,which helps to provide a baseline for future work in thedomain.(iv) We also provide a benchmark dataset for disaster anal-ysis applications covering images from eight differenttypes of natural disasters.The rest of the paper is organized as follows: Section3 discusses the related work. Section 4 provides the back-ground and reviews concepts of the active learning tech-niques. In Section 5 and 6 provide details of the proposedmethodology and dataset, respectively. The details of the ex-perimental setup, experiments and results are provided inSection 7. Finally, Section 8 concludes this study.
In recent years, disaster analysis of images shared on so-cial media outlets received great attention from the research community. Several interesting solutions relying on diversi-fied sets of strategies have been proposed to effectively uti-lize the available information. A majority of the efforts inthis regard rely on multi-modal information including visualfeatures and meta-data comprised of textual, temporal andgeo-location information [20]. For instance, Benjamin et al.[8] utilized the additional information available in the formof meta-data along with visual features extracted through anexisting deep model; namely, AlexNet, pre-trained on Im-ageNet [11]. Both types of information are then evaluatedindividually and in combination with flood-related imagesobtained from social media. Similarly, the work in [3] alsodemonstrates better results for visual features over textualand other information from meta-data in disaster analysis.The majority of the visual features based frameworks fordisaster analysis rely on existing pre-trained models either asfeature descriptors or the models are fine-tuned on disasterrelated images. To this aim, the existing models pre-trainedon both ImageNet [11] and Places [27] datasets have beenemployed. For instance, in [6], an existing model; namely,VGGNet-16 [23] pre-trained on ImageNet is fine-tuned ondisaster related images for categorization of the images intodifferent categories, such as informative and non-informative,damage severity and humanitarian categories. Ahmad et al.[5] utilized existing models pre-trained on both ImageNetand Places dataset as feature descriptors both individuallyand in different combinations. The authors also evaluate theperformance of several handcrafted visual features extracted.More recently, disaster analysis of images shared on so-cial media has also been introduced as a sub-task in a bench-mark competition; namely, MediaEval for two consecutiveyears. In MediaEval-2017 [8], the task focused on the classi-fication of social media imagery into flood-related and non-flooded images. On the other hand, the task in MediaEval-2018 [10] focused on the identification of passable and non-passable roads in social media images. Majority of the solu-tions proposed for the classification of images into floodedand non-flooded categories in MediaEval-2017 relied on deepmodels (e.g., [3,8,19,7]). For instance, in [3] an ensembleframework relying on several deep models used as featuredescriptors has been proposed. Similar trend has been ob-served in MediaEval-2018 for the identification and classi-fication of passable roads through information available onsocial media, where majority of the methods relied on en-sembles of deep models (e.g., [4,12,26,18,16,10]). For in-stance, in [4] multiple deep models were jointly utilized inan early, late and double fusion manner.In the literature, disaster analysis in images has beenmostly treated as a supervised learning task where classi-fication models are trained on training samples annotatedwith human annotators. Two benchmark datasets, namelyDIRSM [9] and FCSM [10], have been mostly reported in the literature [20]. The datasets provide a limited set of im-ages, which are not sufficient to train deep models. More-over, both datasets cover flood related images, only. We be-lieve active learning techniques could be useful to cover thelimitation of lack of sufficient annotated training data in thedomain. Active learning is a semi-supervised learning technique whichselects the training data it wants to learn from [25]. Selectinggood training samples from the data enables active learningtechniques to perform significantly better with fewer train-ing samples compared to passive learning methods [14]. Inpassive learning methods, a large chuck of the data is ran-domly collected from an underlying distribution for trainingpurposes. The main advantage of active learning over pas-sive learning is the ability to make a decision on the basisof the responses from the previous queries for choosing in-stances from the unlabelled pool of images. In this work, wemainly rely on pool-based sampling methods where samplesare drawn from a large pool of unlabelled samples; namely, u = { x i } ni =1 . An initial training set also known as the seeddenoted as υ = { x i } n i =1 is used the train the initial model, θ , and is populated by picking and annotating the instanceswith y i = { y , y , ...m } from the unlabelled pool of sam-ples, iteratively.In the next subsections, we provide a detailed descrip-tion of the two query techniques (i.e., active learning schemes)used in this work along with the different sampling and dis-agreement methods used by those methods.Uncertainty SamplingUncertainty Sampling is one of the most common and widelyused active learning techniques. With uncertainty sampling,the active learner queries the most uncertain instances (i.e.,the samples for which the learner is least certain how to la-bel). The technique is called uncertainty sampling becauseof its use of posterior probabilities in making decision, and isoften straight forward for probabilistic learning models. Forexample, in case of binary classification, uncertainty sam-pling techniques simply ask for the instance that has a pos-terior probability of being positive around 0.5. For the se-lection of the samples, we employed several variants of thistechnique based on the informativeness measure of the un-labelled instances with three different sampling strategies;namely, (i) least confidence, (ii) margin sampling and (iii)entropy sampling. Next, we provide detailed description ofthose sampling strategies. Least Confidence Query Strategy
This sampling strategy aims to choose the instance from thepool for which the learner has the least confidence about itsmost likely label as shown by equation 1, where x , y and θ represent the sample, the most probably label and the under-lying model, respectively. The strategy is more suitable formulti-class classification. For example, if we have two un-labeled instances; namely, D1 and D2, having probabilities(p1, p2 and p3) with values (0.9,0.09,0.01) and (0.2,0.5,0.3)for class labels A, B and C, respectively, the Least Confi-dence (LC) query strategy selects D2 to be labeled as thelearner is less sure about its most likely label. This exam-ple is illustrated in Figure 1. One way to interpret this querystrategy is that the model selects an instance believed to bemislabeled. LC ( X ) = argmax x − p θ ( y | x ) (1) Least ConfidenceD2 is less likely probablecompared to D1 Marginal samplingFor D2 the differencebetween the top twomost probable classes ishigher than D1Entropy Samplingentropy for D1 is lessthan D2, thus D2 isselected
Fig. 1
An illustration of the working mechanism of the different sam-pling strategies used for uncertainty sampling. The sampling strategies;namely, LC, MS and ES, are represented in red, green and yellow col-ors, respectively. LC and MS consider the top 1 and 2 most probablelabels while ES decides on the basis of the complete probability distri-bution considering all classes.
Margin Sampling
One shortcoming of the LC query strategy is the decisionon the basis of the most probable label only. The LC querystrategy does not consider the rest of the labels which mightbe useful in the selection process. In order to cope with thislimitation, Margin Sampling (MS) incorporates the poste-rior probability of the second most likely label by selectingan instance having the least difference between the top twomost probable labels. Let’s suppose y and y are the top twomost probable labels for a sample x under a model θ . Thenthe margin between the two samples can be represented byequation 2.Considering the previous example presented in Figure1, margin sampling selects D2 as the difference between itstwo most probable labels (i.e., . − . . ) is less thanthe difference between the two most probable labels of D1(i.e., . − .
09 = 0 . ). The low difference between the Naina Said et al. labels of D2 indicates that the instance is ambiguous andthus getting the true label of the instance would help in theclassification process.
M S ( X ) = p θ ( y | x ) − p θ ( y | x ) (2) Entropy Sampling
MS considers the top two most probable labels in the de-cision making process; however, for a dataset with highernumber of class labels, the top two most probable labelsare not sufficient to represent the probability distribution. Tothis aim, the Entropy Sampling (ES) strategy efficiently uti-lizes the probability distribution by calculating the entropyof each instance using equation 3, where P ( y | x ) representsthe posterior probability while H is the uncertainty measureand Y is the output class. Subsequently, an instance with thehighest value is queried. In case of our example shown inFigure 1, D1 yields a value of 0.155 while D2 has a valueof 0.447. Therefore entropy sampling selects the instanceD2 for labelling. In case of binary classification, entropysampling performs as margin and least confident sampling.However, it is most useful for probabilistic multi-class clas-sification problems. ES ( x ) = − X y(cid:15)Y P θ ( y | x ) log P θ ( y | x ) (3)Query By CommitteeThe other active learning technique employed in this workis based on the query by Committee strategy. In this method,a query of different competing hypotheses (i.e., trained clas-sifiers represented as C = ( θ , θ , θ ...θ n ) of the current la-belled data set namely λ is maintained. The queries are thenselected by measuring the disagreement between these hy-potheses. The aim of the query by committee strategy is toreduce the version space, which is the set of hypotheses con-sistent with the current labelled set. For example, if machinelearning is used to search for the best model within the ver-sion space then the aim of the query by committee method isto constrain the size of this space as much as possible lead-ing to a more precise search with as few labelled instancesas possible [22]. In case of several hypotheses, the instanceto be labeled next is chosen by measuring the disagreementamong the hypotheses. Different strategies can be utilized tomeasure the disagreement, in this study we use three differ-ent strategies as detailed below. Vote Entropy
Vote entropy can be considered as query by Committee gen-eralization of the entropy based uncertainty sampling, and is calculated by equation 4, where y i is the vector of all pos-sible labels, C represents the committee of the classifierswhile V ( y i ) is the total number of votes for label y . Sup-pose there are three classifiers (i.e., committee size is 3),three classes [0,1,2] and five unlabeled instances. Then, inorder to calculate the vote entropy, every classifier is firstasked for its prediction for all the unlabelled instances. Sup-pose the predictions returned for a single instance by all thethree classifiers is [0, 1, 0] (i.e., classifier 1 predicts that theinstance lies in class-0, classifier 2 predicts it as a samplefrom class-1 and classifier 3 also predicts it as class-0). Eachinstance has a corresponding probability distribution (i.e.,the distribution of class labels when picking the classifier atrandom). In the stated example, there are two votes for 0,one vote for 1 and 0 votes for 2. Therefore, the probabilitydistribution for this instance is [0. 6666, 0.3333, 0]. Amongall the five instances, vote entropy selects the instance whichhas the largest entropy of this vote distribution. V E ( x ) = arg x max − X i V ( y i ) C log V ( y i ) C (4) Consensus Entropy
In consensus entropy, instead of calculating the probabilitydistribution of the votes, the average of the class probabil-ities provided by each classifier in the committee is calcu-lated. This average class probability is called the consensusprobability. Once the consensus probability is calculated us-ing equation 5 (where C represents the committee of theclassifiers), its entropy is computed and the instance withthe largest entropy is selected to be labelled by the labeler. CE ( x ) = 1 C C X c =1 P θ ( y i ) (5) Max disagreement
The Max disagreement sampling technique calculates thedisagreement of each learner by using the consensus proba-bility and then selects the instance with the largest disagree-ment. In this way, it deals with the issue of the other twostrategies which take the actual disagreement into accountin a weak sense.
Figure 2 provides the block diagram of the proposed method-ology. The framework is composed of three main compo-nents; namely, (i) feature extraction, (ii) collection/annotationof the training samples through active learning and (iii) clas-sification/evaluation. For feature extraction, we rely on an ctive Learning for Event Detection in Support of Disaster Analysis Applications 5 existing pre-trained model. For collection/annotation of thetraining samples, several active learning techniques are uti-lized. The classification phase is based on Support VectorMachines (SVMs). The feature extraction and classificationphases are rather standard, and the main strength of the pro-posed framework stems from the active learning part wherewe collect/annotate relevant training samples from an unla-beled pool of images retrieved from social media outlets. Inthe next subsections, we provide detailed analysis of thosephases.Feature Extraction and classificationFor feature extraction, we rely on an existing deep model,ResNet-50 [13], pre-trained on ImageNet [11]. The modelis used as feature descriptor without any retraining and fine-tuning. The basic motivation for using the existing pre-trainedmodels as feature descriptor comes from our previous work[5,1] where we have shown outstanding generalization capa-bilities on disaster images. Features are extracted from thetop fully connected layer resulting in a 1000 dimensionalfeature vector and the classification phase is based on SVM.Active LearningIn this phase, as a first step, we divide the images collectedfrom social media into two sub-sets; namely, (i) initial train-ing set, which is also known as the seed and is annotatedwith human annotators, and (ii) unlabeled pool of images.An SVM classifier is then trained on the initial small labeledtraining set and the initial accuracy is recorded in the secondstep. The training set is then populated by querying imagesfrom the unlabeled pool of images in step 3, iteratively. Tothis aim, we employed two methods; namely, (i) UncertaintySampling and (ii) Query By Committee. For each method,three different sampling/disagreement strategies are utilizedas described in Section 4. Steps 2 and step 3 are repeated fora given number of iterations as detailed in the experimentalsetup Section 7.
Our new collected dataset covers images from most com-mon types of natural disasters; including, cyclone, drought,earthquake, floods, landslides, thunderstorm, snowstorm andwildfires. The images are downloaded from social mediaplatforms using the corresponding keywords. The collectionof the images is divided into two sub-sets; namely, an ini-tial training set also known as seed and an unlabeled pool ofimages. For our initial training which is the only part of the training set annotated by human annotators, a subset com-posed of 160 images collected for each class/type of disasteris randomly selected and annotated by human annotators in acrowd sourcing study. Similarly, the test set, which is com-posed of 2,516 images, has also been manually examinedand annotated in the crowd-sourcing study. The rest of thecollected images are treated as an unlabeled pool of imagescontaining a large portion of irrelevant images. Moreover,in the comparison against baselines, for one of the methodsas detailed in Section 7, we also manually annotated the un-labeled pool of images resulting in around 2500 additionalannotated images. Figure 3 provides some sample imagesfrom the dataset.
Experimental SetupThe objective of our experiments is manifold. Our objectiveis to analyze the performance of active learning in supportof disaster analysis in images shared on social networks. Wealso aim to analyze the performance of different active learn-ing techniques when using different sampling/disagreementstrategies. Moreover, we want to analyze the difference inthe performances of a model/classifier trained on human an-notated dataset and training samples collected through theactive learning techniques. To achieve those objectives, weperformed the following experiments: – First, we analyze the performance of two commonly usedtechniques of pool-based sampling active learning; namely,uncertainty sampling and query by committee. – Then, we investigate the impact of using different sam-pling and disagreement strategies in conjunction withactive learning methods on their overall performance. – Finally, we assess and evaluate the performance of ac-tive learning techniques against two baseline methodswhere one of the fully supervised classifiers is trainedon labeled data annotated by human annotators while theother is trained on the complete pool of images that in-cludes irrelevant ones.We used the same experimental setup for all our exper-imental studies. Specifically, our initial training set (seed),annotated manually, is composed of 160 images covering 20samples from each of the eight different types of natural dis-asters. Moreover, we used a different number of iterations(max 2000) in our experiments. In each iteration, a singleimage from the pool of unlabelled images is included in thetraining set.
Naina Said et al.
Initial Trainingset LearningModel Evaluation ontest setunlabelledpool/dataactive queryselectionDeepFeatures(ResNet)
Input Images Feature Extraction Active Learning Block Evaluation on test set
Fig. 2
Block diagram of the proposed methodology.
Fig. 3
Sample images from the dataset.
Experimental resultsTable 1 provides the evaluation results of the uncertaintysampling method with three different sampling strategies;namely, LC, MS and ES using a variable number of itera-tions ranging from 1 to 2000 (step size of 250). As expected,the accuracy improves by adding relevant samples from theunlabeled pool of images to the initial training set in eachiteration until the accuracy stabilizes for all three methods.Here one important observation is the variation in the per-formances of the method with the three different samplingtechniques as the LC considers only the most probable la-bel, MS considers the top two while ES makes use of allthe labels in it decision of choosing a sample from the pool.No significant difference was observed when the number ofiterations is around 2000. However, higher variations wereobserved in the accuracy of the different sampling strate-gies when the number of iterations is below 1000. At be- ginning, surprisingly, MS and LC strategies performed wellcompared to ES, which shows the importance of the makeuse of most probably labels only in the decision making pro-cess. However, relying on the most probably label increasesdependence on the accuracy of the initial model/classifiertrained on the initial small training set.
Table 1
Evaluation of the different sampling strategies for uncertaintysampling based method at different number of queries.
Sampling strategy Accuracy (%) at different queries
In Table 2, we provide the experimental results of queryby committee based active learning method with differentdisagreement strategies given a number of iterations. Over-all, better accuracy is obtained compared to the uncertaintysampling methods, which is mainly due to employing sev-eral hypotheses/models in the sample selection process. Asfar as the comparison of the disagreement strategies is con-cerned, slightly better results are observed for the CE andMD strategies compared to the VE.In order to better analyze the variations in the accuracyof these methods with different sampling and disagreementstrategies at different iterations, Figure 4 provides the per-formance of the methods with different sampling and dis-agreement strategies at each iteration. As can be seen, boththe methods start at lower accuracy with all sampling anddisagreement strategies and improve iteratively. Compared
Table 2
Evaluation results of the different disagreement strategiesused for uncertainty sampling at different number of queries. disagreement strategy Accuracy (%) at different queries ctive Learning for Event Detection in Support of Disaster Analysis Applications 7
Fig. 4
Comparison of both methods with different sampling strategies
Fig. 5
Comparisons of the active learning methods against baseline. to uncertainty sampling, the cures are more smoother forquery by committee method. Moreover, the accuracy im-proves more rapidly and achieves stability sooner (i.e., after1000 iterations the accuracy is stabilized).The main focus of the paper is to analyze and evaluatethe importance/application of active learning techniques indisaster analysis and to show how the active learning compo-nent can further improve the performances of disaster anal-ysis frameworks with less annotated data. Thus, in orderto show the effectiveness of the active learning methods,instead of sate-of-the-art methods, we compare the resultsagainst two extreme cases reported as baseline 1 and base-line 2 as shown in Figure 5. In the first baseline method, anSVM is trained on human annotated training set, where rel-evant samples were collected and annotated by human ob-servers from the pool of images. In the experiment, featuresare extracted with the same deep model (i.e., ResNet) us-ing the same parameters for the SVM classifier. Moreover,a significant amount of training samples (i.e., around 2500)have been used for training the classifier. In the second case,we trained an SVM classifier on the complete pool of im-ages without removing the irrelevant images with the aimto analyze how much the irrelevant images affect the per-formance of the classifier. As can be seen in most of thecases the active learning methods have comparable resultsto those obtained from the baseline 1 with fully supervisedmethod, which uses a human annotated training set. Thoseresults illustrate the effectiveness of the active learning tech-niques where a small annotated dataset is utilized to obtainbetter results without involving human annotators in the te-dious job of annotation large training sets. In the secondcase, the accuracy has been reduced significantly showingthe efficacy of the active learning techniques able to pickright samples for training among the pool of images.
Lessons learned
The lessons learned from the experiments are: – The accuracy improves by adding relevant samples fromthe unlabeled pool of images to the initial training setin each iteration until the accuracy stabilizes at certainpoint. – Better accuracy against the baseline methods illustratesthe effectiveness of the active learning techniques wherea small annotated dataset is utilized to obtain better re-sults without involving human annotators in the tediousjob of annotation large training sets.
In this paper we presented an active learning approach forthe disaster analysis in images shared on social media out-lets. We mainly used two techniques with several samplingand disagreement strategies for each of the methods. Our ex-perimental results illustrate the effectiveness of using activelearning techniques and their ability to produce results com-parable to those obtained using human annotated trainingsets. Our experimental results also illustrate that the classifi-cation accuracy improves with the inclusion of images fromthe unlabelled pool of images in each iteration using activelearning. Furthermore, our proposed iterative technique ulti-mately achieves stability in terms of classification accuracythrough the progressive inclusion of images from the unla-belled pool of images. Finally, it has been demonstrated thatthe query by committee active learning method is more ef-fective for the disaster analysis in images compared to theuncertainty sampling based active learning methods.
References
1. Ahmad, K., Conci, N.: How deep features have improved eventrecognition in multimedia: A survey. ACM Transactions on Mul-timedia Computing, Communications, and Applications (TOMM) (2), 39 (2019)2. Ahmad, K., Mekhalfi, M.L., Conci, N.: Event recognition in per-sonal photo collections: An active learning approach. ElectronicImaging (2), 1–5 (2018)3. Ahmad, K., Pogorelov, K., Riegler, M., Conci, N., Halvorsen, P.:Social media and satellites. Multimedia Tools and Applicationspp. 1–39 (2018)4. Ahmad, K., Pogorelov, K., Riegler, M., Ostroukhova, O.,Halvorsen, P., Conci, N., Dahyot, R.: Automatic detection of pass-able roads after floods in remote sensed and social media data.Signal Processing: Image Communication , 110–118 (2019)5. Ahmad, K., Sohail, A., Conci, N., De Natale, F.: A comparativestudy of global and deep features for the analysis of user-generatednatural disaster related images. In: 2018 IEEE 13th Image, Video,and Multidimensional Signal Processing Workshop (IVMSP), pp.1–5. IEEE (2018) Naina Said et al.6. Alam, F., Ofli, F., Imran, M.: Processing social media images bycombining human and machine computing during crises. Inter-national Journal of Human–Computer Interaction (4), 311–327(2018)7. Avgerinakis, K., Moumtzidou, A., Andreadis, S., Michail, E., Gi-alampoukidis, I., Vrochidis, S., Kompatsiaris, I.: Visual and tex-tual analysis of social media and satellite images for flood detec-tion@ multimedia satellite task mediaeval 2017. In: Proceedingsof the Working Notes Proceeding MediaEval Workshop, Dublin,Ireland, pp. 13–15 (2017)8. Bischke, B., Bhardwaj, P., Gautam, A., Helber, P., Borth, D., Den-gel, A.: Detection of flooding events in social multimedia andsatellite imagery using deep neural networks. In: Working NotesProceedings MediaEval Workshop, p. 2 (2017)9. Bischke, B., Helber, P., Schulze, C., Venkat, S., Dengel, A., Borth,D.: The multimedia satellite task at mediaeval 2017: Emergenceresponse for flooding events. In: Proceedings of the MediaEval2017 Workshop (Sept. 13-15, 2017). Dublin, Ireland (2017)10. Bischke, B., Helber, P., Zhao, Z., de Bruijn, J., Borth, D.: The mul-timedia satellite task at mediaeval 2018: Emergency response forflooding events. In: Proceedings of the MediaEval 2018 Work-shop. Sophia-Antipolis, France (Oct. 29-31, 2018)11. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Im-agenet: A large-scale hierarchical image database. In: ComputerVision and Pattern Recognition, 2009. CVPR 2009. IEEE Confer-ence on, pp. 248–255. Ieee (2009)12. Feng, Y., Sester, M.: Extraction of pluvial flood relevant volun-teered geographic information (vgi) by deep learning from usergenerated texts and photos. ISPRS International Journal of Geo-Information (2), 39 (2018)13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning forimage recognition. In: Proceedings of the IEEE conference oncomputer vision and pattern recognition, pp. 770–778 (2016)14. Käding, C., Rodner, E., Freytag, A., Mothes, O., Barz, B., Denzler,J., AG, C.Z.: Active learning for regression tasks with expectedmodel output changes. In: BMVC, p. 103 (2018)15. Karlos, S., Kanas, V.G., Fazakis, N., Aridas, C., Kotsiantis, S.:Investigating the benefits of exploiting incremental learners un-der active learning scheme. In: IFIP International Conference onArtificial Intelligence Applications and Innovations, pp. 37–49.Springer (2019)16. Kirchknopf, A., Slijepcevic, D., Zeppelzauer, M., Seidl, M.: De-tection of road passability from social media and satellite im-ages. In: Proceedings of the MediaEval 2018 Workshop. Sophia-Antipolis, France (Oct. 29-31, 2018)17. Liu, Y., Li, Z., Zhou, C., Jiang, Y., Sun, J., Wang, M., He, X.:Generative adversarial active learning for unsupervised outlier de-tection. IEEE Transactions on Knowledge and Data Engineering(2019)18. Moumtzidou, A., Giannakeris, P., Andreadis, S., Mavropoulos, A.,Meditskos, G., Gialampoukidis, I., Avgerinakis, K., Kompatsiaris,I.: A multimodal approach in estimating road passability through aflooded area using social media and satellite images. In: Proceed-ings of the MediaEval 2018 Workshop. Sophia-Antipolis, France(Oct. 29-31, 2018)19. Nogueira, K., Fadel, S.G., Dourado, Í.C., de Oliveira Werneck,R., Muñoz, J.A., Penatti, O.A., Calumby, R.T., Li, L., dos Santos,J.A., da Silva Torres, R.: Data-driven flood detection using neuralnetworks. In: MediaEval (2017)20. Said, N., Ahmad, K., Regular, M., Pogorelov, K., Hassan, L., Ah-mad, N., Conci, N.: Natural disasters detection in social mediaand satellite imagery: a survey. arXiv preprint arXiv:1901.04277(2019)21. Sener, O., Savarese, S.: Active learning for convolutional neuralnetworks: Acore-set approach. stat , 21 (2018) 22. Settles, B.: Active learning literature survey. Tech. rep., Universityof Wisconsin-Madison Department of Computer Sciences (2009)23. Simonyan, K., Zisserman, A.: Very deep convolutional networksfor large-scale image recognition. arXiv preprint arXiv:1409.1556(2014)24. Tuia, D., Volpi, M., Copa, L., Kanevski, M., Munoz-Mari, J.: Asurvey of active learning algorithms for supervised remote sensingimage classification. IEEE Journal of Selected Topics in SignalProcessing (3), 606–617 (2011)25. Zhang, X.Y., Shi, H., Zhu, X., Li, P.: Active semi-supervised learn-ing based on self-expressive correlation with generative adversar-ial networks. Neurocomputing345