#ISIS vs #ActionCountersTerrorism: A Computational Analysis of Extremist and Counter-extremist Twitter Narratives
Fatima Zahrah
Department of Computer ScienceUniversity of OxfordOxford, [email protected]
Jason R. C. Nurse
School of ComputingUniversity of KentCanterbury, [email protected]
Michael Goldsmith
Department of Computer ScienceUniversity of OxfordOxford, [email protected]
Abstract —The rapid expansion of cyberspace has greatlyfacilitated the strategic shift of traditional crimes to on-line platforms. This has included malicious actors, such asextremist organisations, making use of online networks todisseminate propaganda and incite violence through rad-icalising individuals. In this article, we seek to advancecurrent research by exploring how supporters of extremistorganisations craft and disseminate their content, and howposts from counter-extremism agencies compare to them.In particular, this study will apply computational tech-niques to analyse the narratives of various pro-extremistand counter-extremist Twitter accounts, and investigate howthe psychological motivation behind the messages comparesbetween pro-ISIS and counter-extremism narratives. Ourfindings show that pro-extremist accounts often use differ-ent strategies to disseminate content (such as the types ofhashtags used) when compared to counter-extremist accountsacross different types of organisations, including accounts ofgovernments and NGOs. Through this study, we provideunique insights into both extremist and counter-extremistnarratives on social media platforms. Furthermore, we defineseveral avenues for discussion regarding the extent to whichcounter-messaging may be effective at diminishing the onlineinfluence of extremist and other criminal organisations.
Index Terms —Cybercrime, online radicalisation, counter-extremism, social media analysis, Twitter, cyber-criminals.
1. Introduction
The past few decades have demonstrated how theInternet is playing an ever-increasing role in daily life,and has become an integral asset in society. In partic-ular, the use of various digital technologies and onlineplatforms for communication has been rapidly adoptedinto the home and work place alike. However, this hasalso introduced several implications as various maliciousactors, or cyber-criminals, are quickly exploiting both thebenefits afforded by such technologies as well as thevulnerabilities presented by them for their own criminalgains. Digital communities not only bring people closertogether but also, inadvertently, provide criminals withnew ways to access potential victims online. This hasincluded extremist organisations strategically shifting theirradicalisation and recruitment processes to online plat-forms for the purpose of indoctrinating individuals [1]. The same technologies that allow for a globalised worldto interact seamlessly are also being utilised, adapted andabused by extremist organisations to target individuals andensure organisational longevity [2].Shifting to social media and online means of com-munication has also provided the additional benefits ofgranting extremists with a perceived sense of anonymity,allowing access to an increased audience size, and utilisinginteractive features provided by online platforms to facili-tate the acts of like-minded individuals exchanging radicalthoughts [3]. One of the leading examples of an extremistorganisation making use of social media platforms for thepurpose of radicalisation is ISIS. ISIS’ early social mediastrategies on Twitter emphasised several of the uniquecharacteristics of social media listed above. In many waysISIS’ highly energised recruitment efforts online and itsreliance on the Internet have been central to its identity[4], in addition to introducing many initiatives from lawenforcement agencies to monitor and remove offensivecontent online. The UK government specifically outlinedonline hate crime, with particular emphasis on onlineextremism, as one of the principal threats to cyber securityin their National Cyber Security Strategy [5]. Terrorist useof the Internet has also been highlighted as one of theprimary forms of harmful and illegal content online intheir Online Harms Paper [6].Although the development of legislative and policingcapabilities to prevent acts of extremism is clearly re-quired, constructing approaches to reduce the radicalisa-tion effects and impacts of extremist propaganda is alsocrucial to counter them. Such approaches are referred toas countering violent extremism (CVE), and have beenperceived by both researchers and policy makers alike tobeing central to the process of addressing the pressingneed to combat radicalisation to violence and extremism[7]. CVE programs have generally included carrying focusgroups and community engagement programs in particulardemographics to encourage discussion around identity andsocial integration, and, specifically within the UK, theteaching of fundamental British values to deter extremism[8]. More recently, the use of the Internet as an aid in CVEstrategies has become increasingly apparent; for instance,some of the work carried out by Moonshot CVE, a socialenterprise working to “disrupt and eventually end violentextremism” [9], makes use of Internet capabilities to de-velop counter-messaging campaigns and provide onlineinterventions to vulnerable individuals. These are intended a r X i v : . [ c s . C Y ] A ug o carry out counter-extremism interventions through thesame digital channels utilised by extremist groups, so asto reach the same vulnerable audiences. As social mediabecomes more present in daily life, CVE strategies mustalso embrace the same technologies to effectively discreditand nullify extremist groups [2].Within the research landscape, several questions havebeen raised regarding the evaluation of such online CVEstrategies, though very little has been carried out to ex-plore how such initiatives compare against the content andstrategies used by extremist organisations. Some studieshave emphasised the influence that online messages canhave on human behaviours and opinions. For instance,a study conducted by Frischlich et al. [10] shows howpeople can be manipulated into agreeing with extremistviewpoints when under conditions of threat propagated byvarious media. Thus it can be assumed that online CVEcould influence opinions in a similar way, however furtherresearch is required to strengthen this hypothesis.In this article, we seek to advance current research bycomputationally exploring both extremist content foundonline, as well as the CVE strategy of counter-narratives[4] designed to diminish the influence of extremist organi-sations on social media platforms. Specifically, we engagein a two part study that considers firstly, how extremistsand counter-extremist organisations craft content and sec-ondly, how the psychological motivation behind the mes-sages compares between them; thus, our contributions pro-vide novel insight into both sides of online extremism. Inparticular, this study will apply computational techniquesto analyse and compare the behaviour of various pro-extremist and counter-extremist Twitter accounts, and theeffects this could have on influencing human behaviour.Through this research, we hope to understand the extent towhich current CVE counter-extremist strategies on Twitterrelate to extremist content, and identify potential avenuesfor future research on whether such CVE approaches canbe made more effective.The remainder of the report will be structured asfollows. Section 2 will review the current literature onextremist use of the Internet. Section 3 will provide adetailed account of our approach and methodology, in-cluding the datasets and data analysis tools that were used.The results and observations from the analysis of boththe pro-ISIS tweets and the counter-extremism tweets willbe discussed in Section 4. We then conclude and outlineavenues for future work in Section 5.
2. Related Work
The phenomenon of radicalisation through onlineplatforms has been researched extensively over thepast decade by counter-terrorism and cyber security re-searchers. Many previous studies have examined key nar-ratives incorporated by ISIS, as well as some of the majorthemes and components used within their online materialsfor the purpose of recruiting and disseminating propa-ganda. One such study is carried out by El-Badawy et al.[11], which closely examines violent Jihadi propaganda inorder to understand their extremist ideology. The findingsfrom this study showed that common beliefs shared by amajority of the global Muslim community, which may not necessarily be extreme, are frequently used to form sol-idarity with a wider target audience. Moreover, justifica-tions from the Quran, Hadith or from scholarship are alsooften used to resonate with their Muslim audience [11].Similarly Torok [12] provides a qualitative analysis of thesocial media accounts of a number of extremist groups; theresults from this study identify a number of key discursiveschemas, and highlight common themes used by variousextremist Islamist groups, including ‘blaming of the West,unity of Islam, restoring the glory of Islam, and theembracing of death’. These findings also reinforce theobservation that the unity of the wider Muslim commu-nity is a key radicalisation mechanism used to normaliseextremist content and actions.More similar to the research that will be covered in thispaper, numerous studies have made use of computationalapproaches to analyse online extremist content and detectradicalisation. One such study is detailed by Vergani andBliuc in [13], where computational text-analysis toolswere used to analyse the first 11 issues
Dabiq to inves-tigate the evolution of ISIS’ language. The results fromthis provided four key findings: affiliation or achievementplays a major role in motivating collective action of thegroup; ISIS is increasingly adopting emotional tones toincrease influence, including anger and anxiety; ISIS textsexhibit more concern for women; and finally, they aremaking more use of Internet jargon to adapt itself toonline environments and appeal to younger audiences.Another such study was conducted by Fernandez et al.in [14], where they explored how online radical contentcould be detected, not just by searching for key terms andexpressions associated with extremist discourse, but byfurther analysing the contextual semantics of such terms[15]. This provided a more realistic and reliable radicali-sation detection model by helping to discriminate radicalcontent from content that only uses radical terminology,i.e. content simply reporting on events or sharing harmlessreligious rhetoric.Despite the extensive research analysing online ex-tremist content, to our knowledge, there have been fewcarried out to systematically or computationally analysecounter-extremism content currently existing online in asimilar way. This could largely be due to the fact that,at present, there are few existing counter-extremism ini-tiatives online, or at least few that exist on mainstreamsocial media platforms such as Twitter. That being said,some of the work within this line of research includesa report by Ashour, which outlined a broad frameworkconsisting of three major “pillars” that could be used tocounter extremists narratives [16]. The first pillar wasformed from a comprehensive message that dismantlesand counter-argues against every dimension of the extrem-ist narrative, such as the theological and political aspects.Secondly, choosing effective ‘messengers’, who could becredible sources of information, namely former extremistswho have been successfully de-radicalised, would alsobe imperative. Finally, the role of the media is essen-tial to effectively disseminate counter-narrative contentand attract a wider audience is also imperative. Morerecently, Wakeford and Smith [17] reinforce this pointby arguing that it is not enough to simply delegitimiseextremist posts; law enforcement agencies need to learnfrom extremist organisations, investigate and understandhat makes them so influential, and harness this in theirown counter-extremism effortsThe research detailed in this paper will therefore aimto fill the gap currently in this research landscape byproviding more extensive empirical and statistical insightinto the strategies used by pro-extremist users and howextremist content is constructed online. In particular, ourwork focuses on the extent to which the first and third pil-lars described above are currently being used in counter-extremist posts. We additionally bring some understandinginto the psychological motivation behind their posts. Bythis, we specifically refer to how language can be ma-nipulated to influence human behaviour. By comparingthese findings to the content shared by counter-extremismagencies, we provide unique insight into both sides ofthe problem, and also provide avenues for discussionregarding the extent to which counter-messages may beeffective at diminishing the online influence of extremistorganisations.
3. Methodology
Our approach consists of analysing two datasets oftweets—one consisting of tweets from pro-ISIS accountsand the other consisting of tweets from counter-extremismagencies—to gain insight into the linguistic componentsused within them. We first use computational methodsto carry out an empirical analysis to better understandthe techniques used by pro-ISIS supporters and variouscounter-extremism organisations to promote their content.This will include comparing the usage of hashtags, linksto external websites, and the most commonly used terms.Following this, we implement a more comprehensive lin-guistic analysis of the different sets of tweets to gainan understanding of how online narratives are framed.Through applying insights from previous research regard-ing the use of language and motivational theory (such asRegulatory Focus theory introduced by Higgins in [18])in certain texts, this analysis will allow us to furtherexplore how certain linguistic components can be usedto influence behavioural change. This will also provideinsight on whether online counter-narrative content canbe crafted more effectively, for instance, by utilising moreappropriate linguistic terms. Below, we describe the Twit-ter datasets that are used in the study, and the methodsand tools used to analyse them.
In order to analyse extremist content on social media,we acquired a publicly available dataset of noticeablypro-ISIS tweets posted by key ISIS-supporting Twitteraccounts . The dataset was published by the Kaggle datascience community, and consists of over 17,000 English-only tweets retrieved from 112 distinct pro-ISIS supporteraccounts over a period of three months during the after-math of the November 2015 Paris terror attacks. Thesetweets were identified as being pro-ISIS after analysingspecific indicators. This includes using certain key termswithin their username, Twitter bio, or the actual tweetitself; following or being followed by other known radical accounts; or utilising images of ISIS logos or well-knownradical leaders. This particular dataset has been used inseveral previous studies, including [15] and [19]. Ourstudy focussed on English-only tweets as the counter-extremist tweets used in this study were retrieved fromEnglish-speaking organisations only—further details ofthis are given below—making the analysis of the extremistand counter-extremist tweets more comparable.Before using this dataset in our analysis, we firstvalidated that these tweets were in fact posted by pro-ISIS Twitter accounts by manually checking the profilesof the 112 accounts using the Twitter API. Our assumptionhere is that, if the account no longer exists on Twitter or,in other words, has been blocked from the social mediaplatform, then the account most likely did belong to anISIS-supporting individual or group. This is due to the factthat the suspension or blocking of an account suggests thatit had displayed malicious behaviour that did not complywith the Twitter terms of service. From this, we identifiedthat only two of the Twitter accounts were not blockedand still existed on the platform at the time this researchwas conducted, where one of the accounts belonged to ajournalist, and the other belonged to a researcher focusingon Jihadi groups. These two accounts and any tweetsposted by these accounts were thus deleted from the pro-ISIS dataset, leaving a final total of 16,949 tweets from110 pro-ISIS Twitter accounts.In addition to the pro-ISIS dataset, we used the TwitterAPI to retrieve a number of tweets from the Twitteraccounts of major, English-speaking organisations specif-ically dealing with counter-extremism, including Gov-ernments and Law Enforcement Agencies (GLEAs), andNGOs. The accounts that tweets were retrieved frombelong to the following agencies: The Commission forCountering Extremism in the UK (@CommissionCE);Counter Terrorism Policing UK (@TerrorismPolice); theUK Home Office (@ukhomeoffice); the US Departmentof State Bureau of Counterterrorism (@StateDeptCT);the Counter Extremism Project (@FightExtremism), aninternational policy organisation; the Global Center on Co-operative Security (@GlobalCtr), an international policyorganisation; and Tech Against Terrorism (@techvsterror-ism), an NGO supporting tech industries. These accountswere selected on the basis of the volume of tweets relevantto counter-extremism that were available to retrieve.Additionally, since the Twitter account of the UKHome Office does not solely post counter-extremism con-tent, the tweets retrieved from it were filtered with thecriteria that they include an extremism-related term (e.g., CVE , terrorism , extremist ). To gain deeper insight intohow current counter narratives are constructed, we sepa-rated this collection of counter-extremism tweets into threefurther datasets: counter-extremism tweets from GLEAsin the UK, counter-extremism tweets from GLEAs in theUS, and counter-extremism tweets from NGOs. Each ofthese datasets held between 2,000 and 3,000 tweets (with2481 tweets from UK GLEAs, 2703 tweets from USGLEAs, and 2649 tweets from NGOs) and were anal-ysed separately to investigate whether counter-narrativesare crafted differently in each of the three organisationalbodies. We then created a further dataset with the tweetsfrom all the above mentioned counter-extremism datasets(with 7833 tweets in total) to easier compare the resultsrom extremist and counter-extremist posts.Before analysing the datasets, a series of pre-processing steps were carried out to clean the tweetsand prepare them for further linguistic analysis. Thesesteps included: (1) Removing any duplicate tweets orretweets from the datasets to reduce the levels of noise. (2)Removing all punctuation marks. (3) Removing any URLsfrom tweets. Similar data cleaning methods were used by[15] and [19] prior to working with the dataset. It shouldalso be noted here that account names and usernames werenot used throughout the duration of this analysis—onlythe text used within the actual tweets were linguisticallyanalysed after the specified Twitter accounts were chosenand organised into their appropriate sub-datasets. Our analysis of the extremist and counter-extremisttweets is conducted with particular regards to two researchquestions: • RQ1: How are pro-extremist and counter-extremistmessages constituted and what do they focus onpromoting? • RQ2: How do pro-extremist and counter-extremism Twitter accounts compare in termsof the methods used to disseminate content andthe psychological motivation used within theirtweets?The initial empirical analysis will be used to examinedata associated with each dataset of tweets, including themost commonly used hashtags and terms, and some ofthe ‘topics’ that are associated with them. The results fromthis will then be compared across all five datasets of tweetsto identify any similarities or differences between theirrespective approaches to promoting their content. Thisanalysis will be carried out using the Pandas data anal-ysis library and the ‘Natural Language Toolkit’ (NLTK) provided by the Python programming language.To complement the aforementioned analysis, we usea framework majorly based on the theory of utilisingregulatory focus to influence the thoughts and actionsof a target audience through motivational regulation. Inparticular, this analysis is underpinned by the idea thatregulatory focus distinguishes between two types of moti-vational regulation, promotion and prevention, as detailedby Higgins [18]. Here, a promotion focus places emphasison desires and potential goals, and often views these goalsas hopes and aspirations [20]. In contrast, a preventionfocus places emphasis on potential losses, and tends toview goals as duties and obligations [18].Moreover, the findings of Fuglestad et al. [21] alsosupported the notion that the regulatory focus and theframe of a message could be highly relevant to behaviouralchange, which they exemplified with smoking cessationand weight-loss interventions. Their study showed that apromotion focus could be related to the initiation of be-havioural change (such as quitting smoking and dieting),while a prevention focus predicted the long-term mainte-nance of new, healthy behaviours. Some of these findings were applied to our study to observe if either preventionor promotion regulatory focus were used in the extremistor counter-extremist message frames to radicalise or de-radicalise target audiences respectively. This was assessedusing an analysis framework based on the findings fromVaughn [22], which provided some statistical insight intothe linguistic components used in both prevention andpromotion regulatory focus.To analyse the datasets of tweets, we used the pro-grammatically coded dictionary from the Linguistic In-quiry and Word Count (LIWC) linguistic analysis tool toautomate the process of extracting psychological meaningfrom textual content. A similar approach has been used in[22] and other studies to examine and predict the psycho-logical frames and behaviours of various groups as wellas textual content, for instance, to predict depression [23].LIWC is a widely used tool utilised in lexical approachesfor personality measurement, and statistically analysestextual content based on 81 different categories by cal-culating the percentage of words in the input text thatmatch predefined words in a given category. LIWC is usedin our approach to assess the extent to which each of thedatasets of tweets make use of promotion and preventionregulatory focus, in accordance with an analysis frame-work detailed by Vaughn in [22]. Here, Vaughn specifiesthe LIWC categories that share significant differencesin promotion-focussed text and prevention-focussed text.Further details of this have been provided in Section 4.2.below. The next section will detail the results from ouranalysis, and discuss the insights gained from this.
4. Results and Discussion
Hashtags usedby the pro-ISIS Twitter accounts followed most of theobvious political and extremist interests of radical Islamistas well as ISIS-specific adherents. In all the 16,494 pro-ISIS tweets, a total of 2418 distinct hashtags were de-tected, where 41% of the tweets contained at least onehashtag. The 15 most used hashtags found in the pro-ISIS tweets—as well as those found in the other datasetsof tweets—are summarised in Table 1. The most popularhashtags by a wide margin were and , whichwere used 1577 and 1373 times respectively. Consideringthe fact that most ISIS-related activity was based in Syriaand its surrounding areas, it is no surprise that a majorityof the most common hashtags were related to locationswhere ISIS activity was most prevalent, including Iraqand Aleppo, as well as the states which had the mostimpact on ISIS activity—at the time of data collection—including Russia and the USA. The consistent usage ofsuch hashtags helped amplify the coverage of ISIS-relatednews amongst supporter networks.The tweets from counter-extremism NGOs used atotal of 647 distinct hashtags (where 44% of the tweetscontained at least one hashtag), with the tweets from USGLEAs using a similar number of 605 distinct hashtags(where 57% of the tweets contained at least one hashtag).However, the counter-extremism tweets from UK GLEAsused considerably fewer distinct hashtags, as only 375
ABLE 1. T HE MOST USED HASHTAGS FOUND IN THE PRO -ISIS
AND COUNTER - EXTREMISM TWEETS ARE SUMMARISED BELOW . Pro-ISIS Supp. Counter-extremists NGOs US GLEAs UK GLEAs unique hashtags were detected; however, we also foundthat hashtags were use significantly more often, where76% of the tweets contained at least one hashtag. Thissuggests these tweets were more consistent with theirusage of hashtags to promote their content. In terms ofthose which were most used, similar strategies were usedacross all three counter-extremism datasets. As shown inTable 1, the majority of the top hashtags used in the tweetswere based around counter-terrorism and extremism (e.g., , , , ).A notable observation here is that the counter-extremism tweets from both the NGOs dataset and theUS GLEAs dataset use the most similar hashtags, forinstance, both datasets frequently use hashtags related toISIS, including , and . It should be notedhere that the difference between the use of such hashtagsby the pro-ISIS accounts and these counter-extremismaccounts are that pro-ISIS tweets would use these hashtagsto inform their audience of attacks and made by ISISand to promote their cause, as quoted in the follow-ing tweet: “ .US GLEAs would use such hashtags to inform on the USgovernment’s strategies on dealing with ISIS, as shown inthe following tweet: “The US is dedicated to cutting off . This suggests that a majority oftheir counter-extremist policies were tailored to dealingwith ISIS, since this is the only extremist organisationmentioned. NGOs would use hashtags relating to ISISlargely to promote research analysing ISIS activities andbehaviours, for instance: “Successful terrorist operationshave shifted from a ’tactical bonus’ to ’strategic necessity’for Contrastingly, the tweets from UK GLEAs did notmake frequent use of ISIS-related hashtags. Instead, themost commonly used hashtags were concentrated aroundinforming audiences on how to report acts of terrorism,including , , ,and . Another noteworthy point is thatthe top hashtags used by UK GLEAs were used moreconsistently, compared to the tweets from the other twodatasets; the top hashtag in the UK dataset was used 826times, which is considerably more that those used in theNGOs and US datasets, where the top hashtags were used 160 and 163 times respectively. The next part ofthe empirical analysis included determining which wordsand topics were mentioned the most in each dataset oftweets. The most used word in tweets from the pro-ISISaccounts was
ISIS , with
Syria being the second mostcommon word. Along with the frequent mentioning of
Aleppo , Assad and
Iraq , other common terms included killed , army , breaking , soldiers and attack . Further detailsof the most common words in each dataset are providedin the word clouds in Figure 1. A topic model, using theNon-Negative Matrix Factorization (NMF) topic detectionmodel, also provided useful insight into the most discussedsubjects amongst pro-ISIS users as well as key emergingthemes of ISIS ideologies, with the top 15 terms of eachtopic being detailed in Table 2. We found that using theNon-Negative Matrix Factorization (NMF) topic detectionmodel worked better with shorter texts, such as tweets,than other models, like Latent Dirichlet Allocation (LDA)[24]. A large proportion of the most discussed topicsrevolved around reporting the latest reports of attacksagainst ISIS as well as those instigated by ISIS. Thisincludes Topic “Fighting Khawarij is greatest Jehad, whoever iskilled by them receives the reward of a double martyr” .The most common words used in the counter-extremism tweets were similar across all three datasets,with the words terrorism , extremism , and counterextrem-ism being used most frequently in all sets of tweets.The counter-extremism tweets from both NGOs and USGLEAs also mentioned ISIS or ISIL on many occasions,whereas such terms were not amongst the most commonwords used by UK GLEAs. Tweets from UK GLEAs con-sistently made use of words relating to reporting extremistincidents including report , police , suspicious and action-countersterrorism . In contrast, the tweets from NGOs andUS GLEAs focussed more on informing about terroristincidents and counter-extremism initiatives. igure 1. Word clouds of most commonly used words in each dataset: pro-ISIS tweets (top-left), NGOs tweets (top-right), US GLEAs tweets(bottom-left), and UK GLEAs tweets (bottom-right). The topic model provided further insight into whatcontent the counter-extremism tweets promoted in each ofthe three datasets. Again, tweets from both the NGOs andUS GLEAs discussed similar topics. The majority of thesetopics were focussed on new counter extremism effortsand policies, and reports of threats in various countries.Additionally, both sets of tweets discuss the activities ofspecific terrorist organisations, namely ISIS or ISIL, andstrategies to counter them. The tweets from NGOs alsomentioned white supremacists and their online presence,referring to other groups of extremists aside from radicalIslamist organisations. Aside from this, NGOs would oftenpromote workshops or events organised to discuss andpromote counter-terrorism strategies and policies, hence,one of their frequently discussed topics was to promotetickets for such events, specifically Topic “our advice tothe public on what do if caught in a gun or knife terrorattack. It could keep you, your friends and family safe . Such tweets were often postedmultiple times, showing that UK GLEAs often repeatedtweets to emphasise its importance to their target audience.
Although thereare 81 categories in the LIWC standard dictionary, onlythe categories that were proven in [22] to indicate the useof a regulatory focus were used in our analysis framework,though some observations were also made for LIWCcategories which showed notable differences between thedatasets. Table 3 summarises the results from the LIWCanalysis whilst assessing the extent to which a promotionor prevention focus were used. The table shows the meanpercentage of all the words used within the tweets thatfall into a particular LIWC category. For instance, a meanpercentage of 1.35 for positive emotion words implies that1.35% of the words used within the respective datasetwere associated with positive emotion. Example words ofeach LIWC category are also included in the table.
ABLE 2. A
TOPIC MODEL OF THE MOST DISCUSSED TOPICS IN EACH DATASET . Pro-ISIS Supporters Counter-extremists NGOs US GLEAs UK GLEAs
Topic kill, soldier, today,airstrike, civilian,militant, wound, injure,bomb, Russian,children, yesterday,Turkish, dozen, Iraqi Twitter, follow, status,find, visit, ISIS, use,social, media, discuss,internet, join, event,launch, watch status, discuss,workshop, role,present, panel,participate, event, join,host, policies, brief,look, secure, first icymi, yesterday,remark, ISIL, video,testimonies, destroy,strategies, coordinate,global, discuss, effort,envoy, countries,statement extremist, content,terrorist, see, online,via, report, internet,social, media, act,remove, material,access, combat
Topic islam, state, fighter,capture, force, via, unit,fight, group, takfir,declare, call, war,muslim, martyredom icymi, yesterday,remark, ISIL, video,testimonies, destroy,strategies, coordinate,global, discuss, effort,envoy, countries,statement content, extremist,online, platform,facebook, media,social, white, youtube,group, nazi, video,remove, supremacist,hate design, terrorist,special, global, foreign,organise, individual,leader, yesterday,announce, member,entities, group, case,today run, hide, tell, attack,safe, rare, advice,knife, gun, keep,simple, terror, prepare,weapon, firearm
Topic
Al Qaeda, sheikh,jabhat, leader, sham,jaish, release, ibn,jaysh, today, new,village, airstrike,Baghdadi, area counter, terror,extreme, violent,police, UK, global,effort, right, discuss,prevent, support,coordinate, threat, ct attack, kill, people,ISIS, bomb, Taliban,claim, Afghanistan,suicide, soldier, boko,haram, target, group,wound read, latest, via, initial,program, article,policies, safe, remark,foreign, counterterror,recruit, prison, radical,challenge report, suspicious, act,something, could,behaviour, anonymous,live, see, save, instinct,ignore, online, public,vigilant
Topic
Allah, may, accept,brother, protect, one,pleas, make, jazak,victorious, Muslim,sake, reward, love,bless report, online, see,content, suspicious,extremist, help, act,via, terrorist, presence,active, visit, behaviour,find report, new,recommend, juvenile,offend, policies, brief,violent, effort,rehabilitate, prevent,develop, need,societial, program violent, counter,extreme, effort, fact,global, coalition,summit, prevent, build,support, extremist,local, terror, partner presence, help, online,report, via, content,step, find, visit, us,button, speak, get,advice, website
Topic
ISI, US, Assad, fight,Muslim, support, rebel,Syrian, Mosul, help,want, group, back,Aleppo, YPG run, hide, tell, safe,could, attack, rare,remember, simple,keep, knife, gun, terror,watch, weapon counter, terror, global,extreme, forum,violent, internet,prevent, un, effort,strategies, threat,present, launch, nation attack, terrorist,condemn, statement,us, honor, victim, kill,remember, unit, bomb,year, die, mark, ago,families game, secure, enjoy,plan, weekend, great,safe, time, address,stay, go, stadium, look,listen, check
Overall, counter-extremism tweets make use of a pro-motion focus more than the pro-ISIS tweets. Counter-extremism content tended to use the most language as-sociated with positive emotion , which was specified as anindicator for descriptions of pursuing hopes, and thereforeassociated with a promotion focus. Similarly, counter-extremism tweets made use of words related to work , achievement and leisure more than pro-ISIS tweets, whichalso indicated a stronger promotion focus. Further still,such tweets from UK GLEAs had a stronger promotion fo-cus than any of the other counter-extremism organisations.The results from the LIWC analysis also clearly show thatthe pro-ISIS tweets had a very small mean percentage forthe number of words used that were associated with any ofthe four defining conditions for content with a promotionfocus. This suggests that extremists do not tend to frametheir messages around promotion, or view their goals ashopes and aspirations.When looking at the results for the prevention focussection of the LIWC analysis, we can observe, inter-estingly, that counter-extremism tweets posted by UKGLEAs also made use of language associated with aprevention focus more than any of the other sets of tweets.The results for the UK GLEAs showed a greater mean per-centage for almost all of the distinctive LIWC categoriesindicating the strong use of prevention-focussed content.Pennebaker’s findings in [29] found that, when comparedto descriptions of pursuing hopes, descriptions of pursuingduties were more likely to include stories about dynamicsocial interactions and processes. This also infers that ahigher percentage of function words including pronouns , prepositions , auxiliary verbs , negations , and conjunctions are used in the text, which can be observed from Table 3.In addition to this, messages from a prevention focustend to focus on the avoidance of negative outcomes,and because of this, often use more language associatedwith negative emotions . The results show that UK GLEAtweets had a higher percentage for this particular LIWCcategory as well, supporting the observation that it madeuse of a prevention focus the most compared to theother datasets. On the other hand, the pro-ISIS tweetsgenerally seemed to use prevention-focussed narrativesless than counter-extremism tweets, and so did not put asmuch emphasis on duties and obligations as the counter-extremism tweets did.Overall, the results from this linguistic analysis showthat the pro-ISIS tweets used much less regulatory fo-cus, whether promotion or prevention, in their narrativesthan the counter-extremism tweets. Further analysis wouldtherefore be required to assess their radicalisation tech-niques. However, an observation that is common acrossthe results for all five datasets is that they all, gener-ally, used a prevention focus in their messages than apromotion focus. This could suggest that both extremistand counter-extremist narratives view their goals moreas duties and obligations than hopes and aspirations.Through knowledge gained from previous studies suchas [18] and [20], we can infer that using such a focuscould be useful to maintain behavioural change, though itdoes not necessarily inspire initial behavioural change asnarratives from a promotion focus would do. Due to this,it could be beneficial for online counter-extremism nar-ratives to make use of more promotion-focussed contentand pursue positive end-states or goals in order to initiate ABLE 3. R
ESULTS FROM THE
LIWC
ANALYSIS WHEN OBSERVING REGULATORY FOCUS . LIWC Cat-egories Examples Pro-ISIS Supporters Counter-extremists NGOs US GLEAs UK GLEAsPromotion Focus
PositiveEmotion happy, pretty, good 1.35 2.46 2.24 2.22 3.78
Work work, class, boss 1.08 5.68 4.08 3.75 4.72
Achievement try, goal, win 0.97 1.69 2.04 2.50 2.23
Leisure house, TV, music 0.57 2.57 0.82 1.08 1.36
Prevention Focus
Functionwords it, to, no, very 23.97 27.35 26.77 25.95 37.10
Pronouns
I, them, itself 4.29 5.04 2.98 3.65 8.79
PersonalPronouns
I, them, her 2.61 2.88 1.54 2.06 5.14
Conjunctions but, whereas 2.06 3.11 2.67 2.30 4.51
Negations no, never, not 0.64 0.32 0.20 0.26 0.53
NegativeEmotion hate, worthless, enemy 2.63 3.60 3.60 3.70 4.08
Social Pro-cesses talk, us, friend 4.95 9.24 5.78 5.53 11.14
Family mom, brother, cousin 0.24 0.13 0.16 0.12 0.11
Friends pal, buddy, coworker 0.06 0.23 0.27 0.30 0.17 de-radicalisation, although prevention-focussed narrativeswould still be necessary to maintain these efforts.
In addition to exploringthe usage of regulatory focus, LIWC was also used to anal-yse each of the datasets for any other notable distinctionsin the linguistic composition of the tweets. Significantobservations have been summarised in Table 4. Whilstconducting this analysis, an immediate distinction thatcan be seen is the usage of pronouns in each dataset.Overall, all of the counter-extremism datasets generallyused less singular first-person pronouns (such as I , me , my ), and more plural first-person pronouns (such as we , our , us ) than the pro-ISIS tweets. Second-person pronouns(such as you , yours , yourself ) were present in the tweetsfrom UK GLEAs significantly more than any of theother datasets of tweets. This is in line with the previ-ous observations made from the empirical analysis wherethe counter-extremism tweets from UK GLEAs mainlyaddressed their audience directly to inform them of howto properly report and protect against terrorist incidents.When looking at third-person pronouns (such as she , he , they ), the results from the LIWC analysis show that theywere used more in the tweets from the pro-ISIS accountsthan any of the counter-extremism tweets.The use of pronouns in speech and text has beenstudied extensively in previous works, and has often beenidentified as a discursive tool used to persuade audiences.This effect of persuasion is partly due to the variabilityof the scope of reference of the pronouns used, whichis determined by the audience, who can then interpretwhether they are inclusive or exclusive of them [30],[31]. In particular, the use of personal pronouns such as we , you , our and us is a common persuasive techniqueused in writing to make audiences feel more immediatelyinvolved. The LIWC analysis shows that this particularstrategy is used more in the counter-extremism tweetsthan the pro-ISIS tweets; more specifically, the tweetsfrom the UK GLEAs used such pronouns significantlymore than the other sets of tweets. However, it shouldbe noted here that the pro-ISIS tweets used such second- person pronouns (e.g. you , yours ) more than the counter-extremism tweets from NGOs and US GLEAs.Another noteworthy point here is that making use ofthird-person pronouns in political discourse is a tactic thatcan be used to delineate the level of commitment andinvolvement of an organisation to the statement beingmade [32]. This is used most in the pro-ISIS tweets,largely due to the fact that most of these tweets arefrom pro-ISIS supporters, and likely not ISIS themselves,though the counter-extremism tweets—especially thosefrom GLEAs—were directly from official representativesof the organisations. This shows that, in general, thecounter-extremism agencies were more involved or com-mitted to any future responsibilities declared in the tweets.The results from the LIWC analysis also showedsignificant differences in the use of language associatedwith anxiety (e.g. nervous , afraid , tense ). Generally, thecounter-extremism tweets used more anxiety-related lan-guage than the pro-ISIS tweets from UK GLEAs usingsuch language the most, which is supported by the findingsfrom Vergani and Bliuc in [13]. The use of languagerelated to death (e.g. kill , bury , grave ) was more commonin the pro-ISIS tweets, though this is justifiable consid-ering our analysis showed that themes of martyrdom andthe attacks on ISIS, as well as the findings from Torokin [12], were frequently discussed. Another observationis that the pro-ISIS tweets used more religion-associatedlanguage than the counter-extremism tweets. Tweets fromUS GLEAs and NGOs referred to religion slightly morethan those from UK law GLEAs, though this could largelybe due to the fact that these datasets were shown todiscuss ISIS frequently, as noted earlier in Section 4.1.Recent research, such as the study carried out by El-Said in [33], has shown that a major de-radicalisationstrategy, particularly when countering ISIS narratives, isto involve clerics and scholars to promote authentic re-ligious teachings, and use them to refute misinformedreligious teachings propagated by extremists. This pro-vides a further area of development for counter-extremismcampaigns on social media platforms, where making useof such religious teachings could help to directly counter ABLE 4. A
DDITIONAL OBSERVATIONS MADE FROM THE
LIWC
ANALYSIS . LIWC Category Pro-ISIS Supporters Counter-extremists NGOs US GLEAs UK GLEAs I We You
She/he
They
Anxiety
Religion
Death a significant amount of extremist content online.
Despite gaining some useful insights through thisstudy, certain limitations of our approach could haveimpacted our observations. The first limitation that affectsany research carried out in counter-extremism is that itis very difficult to ethically measure the effectiveness ofcounter-extremism initiatives. This makes it challenging tocome to any concrete conclusions about how such counter-narratives can be improved since there is no efficientway to evaluate them (other than first hand experiencewith individuals impacted). Additionally, measuring theeffect of online extremist or counter-extremist content onbehavioural change is also hard to do with ethically-soundmethodologies, and therefore can mainly be supported bythe findings from previous studies and research, as donein this paper. However, since it is undeniable that onlineextremists played a major role in the radicalisation of theirtarget audiences on mainstream social media, it shouldbe possible for counter-extremism narratives to reach thesame platforms as online extremists, and therefore dis-tribute content that is accessible and influential amongtheir target audiences [34].Another point to note about is that the counter-extremism tweets were gathered from different timeframes than the pro-ISIS tweets. For instance, tweetsfrom UK GLEAs were collected from October 2016 toSeptember 2019, tweets from US GLEAs were collectedfrom March 2013 to September 2019, and tweets fromNGOs were collected from January 2015 to September2019. This wide time span is largely due to the lackof counter-extremism content available on Twitter (aninteresting observation in itself). In our study, we felt thatit was more important to gain a dataset of tweets largeenough to analyse and compare with the results of thedataset of the pro-ISIS tweets (which were all from 2015and 2016). It should be noted here, however, that mostof the counter-extremism accounts started posting morefrequently at around the same time the pro-ISIS tweetswere posted (in 2015 and 2016), which is when ISISsupporters were more prevalent on Twitter [35].
5. Conclusions and Future Work
Up until recently, regulation of the internet againstorganisational crime and extremism in online spaces hasmainly concentrated on disruption efforts. Ultimately, ourwork suggests that perhaps other initiatives, such as CVE,could be used to combat online radicalisation. From this study, we sought to advance the current research in onlineonline extremism and counter-extremism narratives bycomparing the online behaviours of Twitter accounts fromboth extremist and counter-extremist organisations, andassessing how the two sets of messages compare with eachother. To our knowledge, this is the first work to explicitlycompare extremist and counter-extremist content in thisway, whilst also applying psychological motivational the-ory to explore how such posts can influence behaviouralchange in online audiences. Although our study analysesdata from one particular use-case of online radicalisationthrough pro-ISIS tweets, we believe that a similar method-ology could be applied to other use cases of radicalisationusing online platforms to gain further insights into theeffectiveness of counter-extremism strategies.Through performing linguistic analysis on datasetsof tweets from pro-ISIS supporters and various counter-extremism organisations, we found that, oftentimes,counter-extremism tweets from certain agencies—namelyUS GLEAs and NGOs—would promote topics and usehashtags which were also used frequently by pro-ISISsupporters. This included frequent discussion around ISISactivity and use of the hashtag . In contrast, counter-extremism tweets from UK GLEAs seemed to share com-pletely different content when compared to each of theother datasets of tweets. In this case, the majority of theirposts were crafted for the purpose of informing onlineaudiences on how to report or protect themselves againstpossible extremist activity, where specific extremist groupswere rarely referred to. Consequently, most of these postswere not constructed to directly counter extremist contentbeing posted online.In terms of the psychological motivation behind thetweets, with specific regards to Regulatory Focus Theory,we found that counter-extremism tweets generally seemedto use regulatory focus more than the pro-ISIS tweets,with tweets from UK GLEAs using such motivationaltheory the most. An avenue for future work here would beto analyse the pro-ISIS tweets with more advanced meth-ods, frameworks or tools to assess any other radicalisationtechniques used by extremists to radicalise and manipulatetheir audience. Our findings also showed that, overall, bothextremist and counter-extremist tweets used prevention-focussed narratives more than promotion-focussed narra-tives. An area for further study would therefore be to ex-plore whether using more promotion-focussed narrativeswould be an effective counter-extremism strategy.Previous research conducted by Higgins in [18] sug-gested that promotion-focussed narratives inspired ini-tial behavioural change, whereas using a prevention fo-cus could facilitate the maintenance of this behaviouralchange. Thus, another hypothesis that may be worth in-estigating here would be whether the regulatory focusof such online extremism and counter-extremism contentchanged chronologically; did such narratives make useof a promotion focus initially, and then shift to using aprevention focus? Such theories could also be explored onother online platforms, not just Twitter.
References [1] S. Sabouni, A. Cullen, and L. Armitage, “A preliminary radicalisa-tion framework based on social engineering techniques,” in , 2017, pp. 1–5.[2] L. Bertram, “Terrorism, the internet and the social media advan-tage: Exploring how terrorist organizations exploit aspects of theinternet, social media and how these same platforms could be usedto counter-violent extremism.”
Journal for Deradicalisation , 2016.[3] L. S. Neo, L. Dillon, P. Shi, J. Tan, Y. Wang, and D. Gomes, “Un-derstanding the Psychology of Persuasive Violent Extremist OnlinePlatforms,” in
Combating Violent Extremism and Radicalisation inthe Digital Era . IGI Global, jan 2016, ch. Chapter 1, pp. 1–15.[4] K. J. Greenberg, “Counter-Radicalization via the Internet,”
TheANNALS of the American Academy of Political and Social Science
Journal for Deradicalization
Frontiers in Psychology
Security Informatics , vol. 2,no. 1, p. 6, 2013. [Online]. Available: https://doi.org/10.1186/2190-8532-2-6[13] M. Vergani and A.-M. Bliuc, “The evolution of the ISIS’ language:A quantitative analysis of the language of the first year of Dabiqmagazine,”
Sicurezza, Terrorismo e societ`a , vol. 7, jan 2015.[14] M. Fernandez, M. Asif, and H. Alani, “Understanding the Rootsof Radicalisation on Twitter,”
ACM Conference on Web Science(WebSci ’18) , pp. 1–10, 2018.[15] M. Fernandez and H. Alani, “Contextual Semantics for Radical-isation Detection on Twitter,” in
Semantic Web for Social GoodWorkshop (SW4SG) at International Semantic Web Conference2018, 9 Oct 2018, CEUR. , 2018.[16] O. Ashour, “Online De-Radicalization? Countering ViolentExtremist Narratives,”
Perspectives on Terrorism
ISIS Propaganda
American Psychologist ,vol. 52, no. 12, pp. 1280–1300, 1997.[19] M. Nouh, J. R. C. Nurse, and M. Goldsmith, “Understandingthe Radical Mind: Identifying Signals to Detect ExtremistContent on Twitter,” in , 2019. [Online].Available: https://doi.org/10.1109/ISI.2019.8823548[20] J.-A. K. Johnsen, S. M. Vambheim, R. Wynn, and S. C. Wangberg,“Language of motivation and emotion in an internet supportgroup for smoking cessation: explorative use of automated contentanalysis to measure regulatory focus,”
Psychology research andbehavior management
Health psychology :official journal of the Division of Health Psychology, AmericanPsychological Association , vol. 27 3S, pp. S260–70, 2008.[22] L. A. Vaughn, “Contents of Hopes and Duties: A LinguisticAnalysis,”
Frontiers in Psychology
The Secret Life of Pronouns: What Our WordsSay About Us . Bloomsbury Press, 2011.[30] J. Wilson,
Politically Speaking : The Pragmatic Analysis of Po-litical Language , ser. Language in society (Oxford, England).Oxford: Basil Blackwell, 1990.[31] Y.-J. Zupnik, “A Pragmatic Analysis of the Use of Person Deixisin Political Discourse,”
Journal of Pragmatics , vol. 21, no. 4, pp.339–383, 1994.[32] V. Ho, “Strategic Use Of Nouns And Pronouns In Public Discourse:The Case Of The Fine-Tuning Of The Medium Of InstructionPolicy In Hong Kong,”
Pragmatics , vol. 23, no. 1, pp. 51–67, 2013.[33] H. El-Said, “De-Radicalising Islamists: Programmes and theirImpact in Muslim Majority States,” International Centre for theStudy of Radicalisation and Political Violence, Tech. Rep. January,2012. [Online]. Available: http://eprints.soas.ac.uk/13957/[34] O. Ashour,
The De-radicalization of Jihadists: Transforming armedIslamist movements . Routledge, 2009.[35] A. Ceron, L. Curini, and S. M. Iacus, “Isis at its apogee: Thearabic discourse on twitter and what we can learn from thatabout isis support and foreign fighters,”