Generating Fake Cyber Threat Intelligence Using Transformer-Based Models
Priyanka Ranade, Aritran Piplai, Sudip Mittal, Anupam Joshi, Tim Finin
GGenerating Fake Cyber Threat IntelligenceUsing Transformer-Based Models
Priyanka Ranade ∗ , Aritran Piplai ∗ , Sudip Mittal † , Anupam Joshi ∗ , Tim Finin ∗ , ∗ Department of Computer Science & Electrical Engineering, University of Maryland, Baltimore County,Email: { priyankaranade1, apiplai1, joshi, finin } @umbc.edu † Department of Computer Science, University of North Carolina, Wilmington,Email: [email protected]
Abstract —Cyber-defense systems are being developed to auto-matically ingest Cyber Threat Intelligence (CTI) that containssemi-structured data and/or text to populate knowledge graphs.A potential risk is that fake CTI can be generated and spreadthrough Open-Source Intelligence (OSINT) communities or onthe Web to effect a data poisoning attack on these systems. Ad-versaries can use fake CTI examples as training input to subvertcyber defense systems, forcing the model to learn incorrect inputsto serve their malicious needs.In this paper, we automatically generate fake CTI text descrip-tions using transformers. We show that given an initial promptsentence, a public language model like GPT-2 with fine-tuning,can generate plausible CTI text with the ability of corruptingcyber-defense systems. We utilize the generated fake CTI textto perform a data poisoning attack on a Cybersecurity Knowl-edge Graph (CKG) and a cybersecurity corpus. The poisoningattack introduced adverse impacts such as returning incorrectreasoning outputs, representation poisoning, and corruption ofother dependent AI-based cyber defense systems. We evaluatewith traditional approaches and conduct a human evaluationstudy with cybersecurity professionals and threat hunters. Basedon the study, professional threat hunters were equally likely toconsider our fake generated CTI as true.
Index Terms —Cybersecurity, Cyber Threat Intelligence, Arti-ficial Intelligence, Data Poisoning Attack
I. I
NTRODUCTION
Open-source platforms such as social media, the dark web,security blogs, and news sources play a vital role in providingthe cybersecurity community with Cyber Threat Intelligence(CTI). This OSINT based threat intelligence complementssources collected by companies like IBM, Virtustotal or Man-diant, by analyzing malware found in the wild, as well as thatobtained by the Intelligence community. CTI is informationabout cybersecurity threats and threat actors that is sharedwith analysts and systems to help detect and mitigate harmfulevents. CTI can be shared as text or as semi-structured datawith some text fields using formats like Structured ThreatInformation Expression (STIX) [1] and Malware InformationSharing Platform (MISP) [2]. Recent research has shown howtext analysis approaches can be used to transform free textthreat information into more structured forms [3]–[10], andeven be ingested into defensive systems to enable detection[11].Although there are many clear benefits to open-source threatintelligence, addressing and handling misinformation across these platforms is a growing concern. The misinformation riskfor the security community is the possible dissemination offalse CTI by threat actors in an attempt to poison systems thatingest and use the information [12]. In January 2021, GoogleThreat Analysis Group discovered an ongoing campaign thattargets security researchers. Various nation state government-backed threat actors created fake accounts and blog posts withtextual cyberseucrity information on a variety of exploits inan attempt to divert security researchers from credible CTIsources [13]. There is also additional research that suggeststhe possibility of future propagation of fake CTI. Maasberget al. [14] conducted a study of methods in propagating fakecybersecurity news and developed components to categorize it.They did not create fake cyber news, just studied its potentialpropagation. The widespread generation of fake CTI itself isheavily under-explored, and is a key contribution of this paper.The widespread propagation of fake CTI primarily impactscyber analysts who rely on the information to keep up todate with current attack vectors, as well as the cyber defensesystems that ingest the information to take correct mitigationsteps [11]. Next-generation cyber defense systems are nowbeing developed to automatically ingest and extract data fromopen source CTI to populate knowledge graphs, that are thenused to detect potential attacks or as training data for machinelearning systems.Adversaries can use fake CTI as training input to subvertcyber defense systems. This type of attack is commonly knownas a data poisoning attack [15]. Many cyber defense systemsthat rely on this data automatically collect streams of CTI datafrom common sources. Adversaries can post fake CTI acrossopen sources, infiltrating the training corpus of AI-based cyberdefense systems with ease. This fake information will appear legitimate to cyber analysts, but will in reality, have falsecomponents that contradict the real data. As can be seen fromthe examples in Table I, convincing fake CTI can be generatedthat provides incorrect information about the vulnerabilitiesexploited by an attack, or its consequences. This can causeconfusion in analysts on what steps to take to address a threat.In an automated system cyber defense system that is ingestingthe CTI, this can also break the reasoning and learning processaltogether or force the model to learn incorrect inputs to servethe adversaries’ malicious goals. Techniques demonstrated for a r X i v : . [ c s . CR ] F e b pen-source CTI can also be applied for covert data, such asproprietary information belonging to a particular company orgovernment entity. In this scenario, potential attack strategieswill more than likely be categorized as insider threats, andadversaries will be employees looking to exploit internalsystems.In this paper, we generate realistic fake CTI examplesby fine-tuning the public GPT-2 model. Transformer-basedmethods are state-of-the art approaches that aid in detectingand generating misinformation on a large scale with minimalhuman effort [16].Our generated fake CTI was successfully able to confuseprofessional threat hunters and led them to label nearly allof the fake CTI as true. We then also use the generated fakeCTI examples to demonstrate a data poisoning attack on aCybersecurity Knowledge Graph (CKG) and a cybersecuritycorpus.Our work makes three main contributions: • We produce a fine-tuned GPT-2 model that generates fakeCTI text (Section III-B), • We demonstrate a possible poisoning pipeline for infil-trating a CKG (Section IV), and • We present an evaluation and analysis of the fake andreal CTI text (Sections III-C and III-D).II. B
ACKGROUND AND R ELATED W ORK
In this section, we present a background of transformerarchitectures and provide related work in the areas of text gen-eration, misinformation, AI-Based cyber systems, knowledgegraphs, and adversarial machine learning.
A. Transformer Models
Encoder-decoder configurations inspired current state-of-theart language models such as GPT [17] and BERT [18] whichutilize the transformer architecture [19]. Similar to RecurrentNeural Network (RNN) based sequence to sequence (Seq2Seq)models, the transformer encoder maps an input sequence intoan abstract high dimensional space. The decoder transformsthe vector into an output sequence. Unlike its Seq2Seq pre-cursor, the transformer does not utilize any RNN and reliessolely on the attention mechanism to generate sequences.Seq2Seq architectures rely on LSTM cells to process aninput sequence one word at a time. In a transformer model,all input words are processed in parallel. Due to this, the trans-former introduces the concept of a positional encoding in orderto capture word ordering information in the n-dimensionalvector of each word. The encoder and decoder components ofthe transformer also contain a multi-head attention mechanism.This can be shown using the equation below:Attention ( Q, K, V ) (cid:124) (cid:123)(cid:122) (cid:125) Queries,Keys,Values = sof tmax (cid:18) QK T √ d k (cid:19) V Where Q represents queries, K represents keys, and V repre-sents values. The complete description of creating these valueshas been presented by Vaswani et al. [19]. At the start ofthe encoder, let y be the initial sentence representation. As it travels through each layer of the encoder, y gets updated bydifferent encoder layers. The input y is utilized to calculate Q , K , and V in the above equation. Attention is calculatedby taking the transpose of the matrix dot product QK anddividing by the square root of the dimension of the keys √ d k .Lastly, using the attention weights, we find the weighted sumof values V . The decoder attention mechanism operates simi-larly to the encoder, but employs masked multihead attention .A linear and softmax layer are also added to produce the outputprobabilities of each word. In this paper, we focus on the GPT-2 model [20] which exclusively uses decoder blocks. B. Transformer based Use-Cases
Generative transformer models have many use-cases suchas machine translation [21], question-answering [22] and textsummarization [23]. A popular example of a generative trans-former model is OpenAI GPT [17]. In recent years, GPT-2[20] and GPT-3 [24], [25] models have also been developed(At the time of writing this paper, GPT-3 is only accessible bya paywall API, and the model along with its other componentsare unavailable). GPT models across generations differ fromeach other in the sizes of data-sets used and number ofparameters added. For example, the WebText dataset used totrain GPT-2 contains eight million documents.In this paper, we utilize GPT-2 in our experiments. Unla-beled data is used to pretrain an unsupervised GPT model fora generic task.
Fine-tuning the generic pre-trained models isa common method of extending the architectures for morespecific tasks [17]. Lee et al. [26] produced patent claimsby fine-tuning the generic pretrained GPT-2 model with U.S.utility patents claims data. Similarly, Feng et al. [27] fine-tuned GPT-2 on a small set of yelp review data-set and usedit as a baseline model for various augmentation experiments.Transformers have been utilized to both detect and generate misinformation . Misinformation can be generally categorizedas lies, fabricated information, unsupported facts, misunder-standings, and outdated facts and is often used to achieveeconomic, political, or social gain [28]. Vijjali et al. [29] utilizeBERT-based transformers to detect false claims surroundingthe COVID-19 pandemic. Similarly, Zellers et al. [30] alsouse a BERT-based model called Grover, which can detectand generate neural fake news. Their evaluation shows thathuman beings found machine-generated disinformation moretrustworthy than human-written information.
C. AI-Based Cyber Systems and Knowledge Graphs
Next-generation cyber defense systems use various knowl-edge representation techniques such as word embeddings andknowledge graphs in order to improve system inference on po-tential attacks. The use of CTI is an integral component of suchsystems. Knowledge graphs for cybersecurity have been usedbefore to represent various entities [31]–[33]. Open sourceCTI has been used to build Cybersecurity Knowledge Graphs(CKG) and other agents to aid cybersecurity analysts workingin an organization [3]–[10]. Mittal et al. created Cyber-All-Intel and CyberTwitter [3], [5] which utilizes a variety ofig. 1: We collected cybersecurity-related text from several OSINT sources and used it to fine-tune the public GPT-2 model,which generated fake CTI descriptions.knowledge representations such as a CKG to augment andstore CTI.The use of knowledge graphs for cyber-defense tasks hasalso been used in malware analysis tasks [34]–[38]. Piplai etal. [32], [39] create a pipeline to extract information from mal-ware after action reports and other unstructured CTI sourcesand represent that in a CKG. They use this prior knowledgestored in a CKG as input to agents in a reinforcement learningenvironment [40]. We demonstrate the effects of the poisoningattack, by ingesting fake CTI on CKG using a complete CTIprocessing pipeline [31], [32].
D. Adversarial Machine Learning and Poisoning Attacks
Adversarial machine learning is a technique used to subvertML systems by providing deceptive inputs to the model.Adversaries use these methods to manipulate AI-based sys-tem learning in order to alter protected behavior and servetheir own malicious goals [41]. There are several types ofadversarial techniques such as evasion, functional extraction,inversion, and poisoning attacks [15]. In this paper, we focuson data poisoning attack strategies. Data poisoning attacks areexamples of methods that directly compromise the integrity ofthe learning process of an AI-based system by contaminatingthe training data-set. These methods rely heavily on the useof synthesized and/or incorrect input data. AI-based cyberdefense system can potentially include fake data into theirtraining corpus. The attacker dominates future output byensuring the system learns fake inputs and performs poorlyon actual data.One example of such an attack is the VirusTotal poisoningattack demonstrated by the McAfee Advanced Threat Researchteam [42]. This attack compromised several intrusion detectionsystems that ingest VirusTotal data. The attacker createdmutant variants of a ransomware family sample and uploaded the mutants to the VirusTotal platform. Intrusion detectionsystems that ingest VirusTotal data classified the mutant filesas the particular ransomware family. Similarly, Khurana et al.perform credibility checks on incoming CTI. They developa reputation score that is used by systems and analysts toevaluate the level of trust for input intelligence data [12].Duddu et al. survey several methods of using machine learningto model adversary behavior [43].III. M
ETHODOLOGY
In this section we describe our fake CTI generation pipeline.Figure 1, presents the overall approach. We begin by creatinga cybersecurity corpus in Section III-A. The cybersecuritycorpus contains a collection of CTI from a variety of OSINTsources. We then fine-tune the pre-trained GPT-2 model onour cybersecurity corpus (Section III-B). The fine-tuned modelallows us to automatically generate large collections of fakeCTI samples. We then evaluate our model and describe apoisoning attack against a CTI extraction pipeline.
A. Creating a Cybersecurity Corpus
We categorize our CTI collection into three main sources,as shown in Figure 1. We collect security news articles,vulnerability databases, and technical Advanced PersistentThreat (APT) reports. The security news category contains500 articles from Krebs on Security [44] and CyberWire Daily[45]. The vulnerability reports contain 16,000 Common Vul-nerability and Exposures (CVE) records provided by MITRECorporation and National Vulnerability Database (NVD) fromyears 2019-2020 [46]. Lastly, we collect 500 technical reportson APTs from the available APTNotes repository [47].
B. Fine-Tuning GPT-2 on Cyber Threat Intelligence Data
The original GPT-2 model was trained with the WebTextdataset [20]. While the WebText dataset contains some general eneration Input Cyber Threat Intelligence True/Fake
APT41 is astate-sponsoredespionage group APT41 is a state-sponsored espionage group which operates against higher education, travelservices, and news/media firms which provide some indication that the group also tracksindividuals and conducts surveillance.
True CTI
APT41 is a state-sponsored espionage group. The Russian, Moscow-based group is responsible forcapturing and stealing computer network communications and intellectual property. APT41employs social engineering techniques to get around network defenses designed to last foryears, and is capable of compromising and stealing dozens of companies
Fake CTI(generated)
Malicious Domainin SolarWindsHack Turnedinto ‘Killswitch’ Malicious Domain in SolarWinds Hack Turned into ‘Killswitch’. A key malicious domain nameused to control potentially thousands of computer systems compromised via the months-longbreach at network monitoring software vendor SolarWinds was commandeered by securityexperts and used as a “killswitch” designed to turn the sprawling cybercrime operationagainst itself.
True CTI
Malicious Domain in SolarWinds Hack Turned into Killswitch Service. This attack took placelast year and affected hundreds of organizations. The attackers took advantage of thevulnerability in Win32k framework to attack their targets through crafted code.
Fake CTI(generated)
An issue wasdiscovered in theQuiz and SurveyMaster plugin An issue was discovered in the Quiz and Survey Master plugin before 7.0.1 for WordPress.It made it possible for unauthenticated attackers to upload arbitrary files and achieveremote code execution.
True CTI
An issue was discovered in the Quiz and Survey Master plugin 1.2.1 for WordPress. Bysetting a CSRF token, an attacker can take advantage of a vulnerability in the survey.phpadmin panel.
Fake CTI(generated)
TABLE I: Fake CTI Samples produced by our fine-tuned GPT-2 model.cybersecurity text, it greatly lacks fine-grained CTI informa-tion useful to the security community. To address this problem,we fine-tune the general model with the cybersecurity corpusdescribed above. The diverse CTI sources in our corpus givesthe GPT-2 model a variety of examples and the ability toadapt to several aspects of the cybersecurity domain.
Pre-trained transformer-based language models like GPT-2 areeasily adaptable to new domains such as cybersecurity. Insteadof training from scratch and initializing with random weights,we initialize the model with pre-trained parameters. We usedthe publicly released pre-trained GPT-2 model with 117Mparameters. The model consists of 12 layers, 786 dimensionalstates and 12 attention heads.During our training, we divide the corpus in a 35% train andtest split. We set block size as 128, batch size as 64, and learn-ing rate as 0.0001. We utilize the Gaussian Error Linear Unit(GELU) activation function. The GPT-2 architecture shown inFigure1, consists of normalization layers [48], attention layer,a standard feed forward neural network, and a soft-max layer.The feed forward neural network contains 786*4 dimensions.We trained the model for twenty three hours (20 epochs) andachieved a a perplexity value 35.9. Examples of the generatedCTI and more details on our experimentation are given in thenext section.
C. Generating Fake CTI
We use our fine-tuned GPT-2 model to generate fake CTIexamples, three of which are shown in Table I. The generationprocess is initiated with a prompt that is fed as an input tothe fine-tuned GPT-2 model (the first column in Table I). Themodel uses the initial prompt to generate the fake CTI. Thegeneration process is shown in Figure 1. The tokenized promptis passed through a normalization layer, then through the firstblock of the attention layer. The block outputs are also passedto a normalization layer and fed to a feed forward neuralnetwork, which adds an activation function and dropout. Itsoutput is passed through a softmax layer, which obtains the positional encoding of the highest probability word inside thevocabulary.The first sample in Table I, provides information on APTgroup APT41. Given the prompt, “APT41 is a state sponsoredespionage group” , the model was able to form a partially falsenarrative about APT41. APT41 is a Chinese state-sponsoredespionage group, not a Russian group as indicated by themodel. Although this is a false fact, the later part of thegenerated CTI is partially true. Despite some true information,the incorrect nation-state information surrounding APT41 isstill present and adds conflicting intelligence if ingested by anAI-based cyber defense system.In the second example , we provide an input prompt froma Krebs on Security article [49]. The model generated fakeCTI, which states kill switch as an actual service, when inactuality, kill switch refers to the method of disconnectingnetworks from the Internet. In addition, it relates the falseservice to the
Win32k framework. This gives the fake CTIenough credibility and seems true to cyber analysts.Lastly for the third example , we provide an input promptfrom a 2019 CVE record. The model generated the correctproduct, but an incorrect associated version and attack type;the true attack was a remote code execution while the gen-erated attack was privilege escalation. While a remote codeexecution attack can be related to a privilege escalation attackin general, the specific context of using a Cross-Site RequestForgery (CSRF) token to gain access to survey.php is incorrectfor this specific product.
D. Evaluating the generated CTI
We next show that the generated fake CTIs are credible. Weuse two approaches to show this. First, we evaluate the abilityof the fine-tuned model to predict our test data by calculatingthe perplexity score. Next, we conduct human evaluation stud-ies. The study required a group of cybersecurity professionalsand threat hunters to label a collection of generated and actualCTI samples as true or fake. The cybersecurity experience ofhe participants range from 2-30 years (in operational settings),with an average experience of 15 years. The idea is to seeif professionals in the field can separate real CTI from fakeinstances generated by our system.In the context of cybersecurity, human evaluation withpotential real-world users of the fake CTI is more indicativethan traditional methods such as perplexity scores. The mainobjective of generating fake CTI is to mislead cyber analystsand bypass intelligence pipelines that they frequently monitor.If the generated CTI does not possess a high range of mal-formed sentence structure, poor grammar, or incomprehensibletext (obvious mistakes indicating the text was produced bya machine), we can assume it has fair potential to appearreal to analysts. Perplexity is a common method to determine“uncertainty” in a language model, by assigning probabilitiesto the test set. Perplexity is measured as the exponentiatedaverage logarithmic loss and ranges from 0-100. The lower theperplexity score, the less uncertainty exists within the model.The base 117M GPT-2 model we fine-tuned has a perplexityscore of 24 [26]. We ensure the model is not evaluated on textfrom the training set by calculating perplexity on a separatetest set and achieve a calculated perplexity score of 35.9,showing strong ability of the model to generate plausible text.In order to evaluate the potential implications of the gen-erated fake CTI in a real world setting, we conduct a studyacross a group of ten cybersecurity professionals and threathunters . We provided the participants with an assessmentset of both true and fake CTI text samples. Using their ownexpertise, participants labeled each text sample in the corpus aseither true or fake. We created the assessment set by collecting112 text samples of true CTI drawn from various sourcesdescribed in Section III-A. We pre-process the text samplesby truncating them to the first 500 words and eliminatingpartial last sentences. We select the first sentence of eachsample as an initial prompt to the fine-tuned GPT-2 model andgenerate a fake CTI example of no more than 500 words. Wefurther divide the 112 samples (56 true CTI and their generatedfake counterparts) into two separate annotation sets to ensuretrue CTI and direct fake counterparts are not part of thesame annotation task. Therefore, each annotation task included28 samples of true text and 28 non-overlapping samples ofgenerated fake data. We randomize the data in each annotationtask assigned to the participants.Participants worked individually, and labeled each of the56 samples as either true or fake. Participants used their ownjudgement in labeling each sample, and were prohibited to useexternal sources like search engines during the assessment.The results of the study are provided in the confusion matrix.The confusion matrix shows the true positive, false negative,false positive, and true negative rates for 560 CTI samples (in-cluding both true and fake data). Of the total 560 samples thatwere rated, the accuracy (36.8%) was less than chance. Thethreat hunters predicted 52.5% incorrectly (74 true samples Our study protocol was evaluated by UMBC’s IRB and classfied as NotHuman Subjects Research as false and 220 false statements as true) and 47.5% samplescorrectly (206 true samples as true and 60 false statements asfalse). Despite their expertise, the threat hunters were only ableto label 60/280 of the generated samples as fake and found thea large majority (78.5%) of the fake samples as true. Theseresults demonstrate the ability of the generated CTI to confusesecurity experts, and portends trouble if such techniques arewidely used. A c t u a l D a t a Participant LabelsTrue False TotalTrue
False
Total
426 134We further investigated the fake samples that were accu-rately labeled as fake and observed more linguistic errorsin the text than in comparison to the fake samples thatwere labeled as true. Although the majority of the fake CTIcontained entities (such as products and attack vectors) thatwere unrelated to each other, we found if the sentence structuredisplayed little or no linguistic deficiencies, the data was likelylabeled as true. We also noticed sources that lacked substantialcontext were likely labeled as false.The generated fake CTI not only has the ability to misleadcybersecurity professionals, but also has the ability to infiltratecyber defense systems. In the next section, we describe howthe generated fake CTI examples can be used to launch a datapoisoning attack.IV. D
ATA P OISONING USING F AKE
CTIWith the fake CTI examples in Table I we can easilysimulate a data poisoning attack where the fake CTI is used astraining input to subvert knowledge extraction pipelines suchas those described by Piplai et al. [32], Mittal et al. [3], [4],Gao et al. [33], [50], and Arnold et al. [10]. Here an attackercan skillfully position fake CTI on multiple OSINT sourceslike Twitter, Stack Overflow, dark web forums, and blogs.Many of the systems described above include nativecrawlers along with cybersecurity concept extractors, entity re-lationship extractors, and knowledge representation techniquessuch as word embeddings, tensors, and knowledge graphs.These either use keyword-based methodologies or dependon AI tools to collect and process the CTI. Many of thesesystems can be easily tricked into including the fake CTI datain a cybersecurity corpus along with the true CTI. This isespecially possible if the attacker is able to craft the fakeCTI in such a way that it “appears very similar” to true CTI.This fake information will then be ingested by a knowledgeextraction pipeline utilized to create knowledge representationsig. 2: CKG populated with data from legitimate true CTIsources.Fig. 3: The poisoned CKG with additional data (red box)extracted from fake CTI.like, Cybersecurity Knowledge Graphs (CKG). Poisoning acorpus with fake CTI can enable an attacker to contaminatethe training data of various AI systems in order to obtain adesired outcome at inference time. With influence over the CTItraining data, an attacker can guide the creation of AI models,where an arbitrary input will result in a particular output usefulto the attacker.Next, we describe an attack on a popular knowledge rep-resentation technique that involves a CKG [4], [31], [32].As we already have access to a complete CTI processing pipeline that outputs a CKG [32], we choose to demonstratethe effects of the poisoning attack on the CKG. Once the fakeCTI has been represented in a knowledge representation it canbe used to influence other AI systems that depend on theserepresentations. We also discuss the effects of the poisoningattack on the CKG in Section IV-B.
A. Processing fake CTI
A CTI ingestion pipeline described in Piplai et al. [32]and similar systems [10], [33], [50] take a CTI source as aninput and produces a CKG as an output. The CKG containscyber entities and their existing relationships. The first stageis a cybersecurity concept extractor that takes a CTI andextracts various cyber entities. This is done by using a NamedEntity Recognizer (NER) trained on a cybersecurity corpus.The second stage, is a deep-neural network based relationshipextractor that takes word embeddings of cyber entity pairs asan input and identifies likely relationships. This results in anentity-relationship set that can be asserted into the CKG. As arunning example, we use the following fake
CTI text as inputto the extraction pipeline- ‘Malicious domain in SolarWinds hack turned intokillswitch service where the malicious user clicks anicon (i.e., a cross-domain link) to connect the servicepage to a specific target.’
When fake CTI is ingested by the pipeline, the cybersecurityconcept extractor will output classifications that serve theadversaries’ goals. The concept extractor classifies ‘clicksan icon’, ‘connect the service’ as ‘Attack-Pattern’. It alsoclassifies ‘SolarWinds hack’ as a ‘Campaign’. These entitiesare extracted from the fake CTI potentially poisoning theCKG.The relationship extractor while processing the fake CTIabove, outputs the following relationships: • ‘Solarwinds hack’ (Campaign)- uses - ‘clicks an icon’(Attack-Pattern). • ‘Solarwinds hack’ (Campaign)- uses - ‘connect the ser-vice’ (Attack-Pattern).The extracted entity relationship set can then be asserted inthe CKG. Figures 2 and 3, describe the state of the CKG before and after asserting knowledge extracted from fakeCTI. Figure 2, contains entities and relationships extractedfrom true CTI samples describing the campaign ‘SolarWindshack’. We can see entities like ‘Orion Software’, identifiedas ‘Tool’, and ‘malicious code’ identified as ‘Attack-Pattern’.These entities are used by the malware in the ‘SolarWindshack’ and are present in the true CTI. We also see ‘simplepassword’ as a vulnerability. Figure 3, contains additionalinformation extracted from fake CTI generated by our model.These additional entities and relationships have been assertedalong with the entity ‘SolarWinds hack’, and are demarcatedby the red box. In this figure, we can see additional ‘Attack-Patterns’ like, ‘connect the service page’ and ‘clicks an icon’being captured in the CKG. These entities have been extractedusing the pipeline from the fake CTI and are an evidence ofow a poisoned corpus with fake CTI can be ingested andrepresented in a CKG.
B. Effects of fake CTI ingestion
The objective of creating a structured knowledge graphfrom the unstructured CTI text is to aid security professionalsin their research. The security professionals can look uppast knowledge about cyber incidents, perform reasoning, andretrieve information with the help of queries. However, ifgenerated fake information is ingested by the CKG as partof a data poisoning attack, it can have detrimental impactssuch as returning wrong reasoning outputs, bad security alertgeneration, representation poisoning, model corruption, etc.For example, if a security professional is interested inknowing which attack campaigns have used ‘click-baits’, theywill be misled by the result ‘Solarwinds hack’. As the fakeCTI has been ingested and represented in the knowledgerepresentation (See Section IV-A). The following SPARQL[51] query when executed on the CKG,
SELECT ?x WHERE {?x a CKG:Campaign;CKG:uses CKG:clicks_an_icon.} will result in the following value:
Solarwinds_hack
If security professionals are interested to know more informa-tion about ‘Solarwinds-hack’, they may also receive incorrectinformation after executing appropriate SPARQL queries.
SELECT ?x WHERE {?x a CKG:Attack-Pattern;ˆCKG:uses CKG:Solarwinds-hack.}
This query results in the following values: malicious_code, offloading_sensitive_tools,connect_the_service_page, clicks_an_icon
Although we obtained some true results (sourced from trueCTI), the presence of fake CTI guided results like, ‘connectthe service page’ and ‘clicks an icon’ have the potential tomislead security professionals. Security professionals modelcybersecurity attacks and generate network/system detectionrules using past available information on the same attacksor similar attacks. They also use these representations togenerate alerts for future attacks. For example, a ‘supply chainattack’ exploiting a ‘small password’ vulnerability ‘offloadingsensitive tools’ may mean that a new variant of the SolarWindshack has surfaced. However, if prior knowledge contains fakeCTI about the same attack, incorrect alerts can be generated.Once these knowledge representations are poisoned, addi-tional defense systems can also be adversely impacted by fakecybersecurity information. For example, many of the insightsgenerated by knowledge graphs are useful to other systemslike AI-based intrusion detection systems [35], [36], [52], oralert-generators [3], [33], reaching a larger breadth of linkedsystems and cybersecurity professionals.V. C
ONCLUSION & F
UTURE W ORK
In this paper, we automatically generated fake CTI textdescriptions by fine-tuning the GPT-2 transformer using a cybersecurity corpus rich in CTI sources. By fine-tuning theGPT-2 transformer with cybersecurity text, we were able toadapt the general model to the cybersecurity domain. Given aninitial prompt, the fine-tuned model is able to generate realisticfake CTI text examples. Our evaluation with cybersecurityprofessionals shows that generated fake CTI could easilymislead cybersecurity experts. We found that cybersecurityprofessionals and threat hunters labeled the majority of thefake CTI samples as true despite their expertise, showing thatthey found the fake CTI samples believable.We use the fake CTI generated by the fine-tuned GPT-2model to demonstrate a data poisoning attack on a knowledgeextraction system that automatically ingests open sourced CTI.We exemplify the impacts of ingesting fake CTI, by comparingthe state of the CKG before and after the data poisoning attack.The adverse impacts of these fake CTI sourced assertionsinclude wrong reasoning outputs, representation poisoning,and model corruption.In ongoing work, will are exploring defences against suchdata poisoning attacks. One approach is to develop systemsthat can detect linguistic errors and disfluencies that generativetransformers commonly produce, but humans rarely make.A second approach to detecting fake CTI text can use acombination of novelty, consistency, provenance, and trust.CTI sources can be given a score that indicates how muchtrust the user wants to put in their information.A
CKNOWLEDGEMENT
This work was supported by a U.S. Department of Defensegrant, a gift from IBM research, and National Science Foun-dation grant
EFERENCES[1] Oasis group. Stix 2.0 documentation. https://oasis-open.github.io/cti-documentation/stix/, May 2013.[2] Cynthia Wagner, Alexandre Dulaunoy, G´erard Wagener, and AndrasIklody. Misp: The design and implementation of a collaborative threatintelligence sharing platform. In
Workshop on Information Sharing andCollaborative Security , pages 49–56. ACM, 2016.[3] Sudip Mittal, Prajit Das, Varish Mulwad, Anupam Joshi, and TimFinin. Cybertwitter: Using twitter to generate alerts for cybersecuritythreats and vulnerabilities.
IEEE/ACM Int. Conf. on Advances in SocialNetworks Analysis and Mining , pages 860–867, 2016.[4] Sudip Mittal, Anupam Joshi, and Tim Finin. Cyber-all-intel: An AI forsecurity related threat intelligence. arXiv:1905.02895 , 2019.[5] Sudip Mittal, Anupam Joshi, and Tim Finin. Thinking, fast and slow:Combining vector spaces and knowledge graphs. arXiv:1708.03310 ,2017.[6] Lorenzo Neil, Sudip Mittal, and Anupam Joshi. Mining threat intel-ligence about open-source projects and libraries from code repositoryissues and bug reports. In
Intelligence and Security Informatics . IEEE,2018.[7] Priyanka Ranade, Sudip Mittal, Anupam Joshi, and Karuna Joshi. Usingdeep neural networks to translate multi-lingual threat intelligence. In
International Conference on Intelligence and Security Informatics , pages238–243. IEEE, 2018.[8] Priyanka Ranade, Sudip Mittal, Anupam Joshi, and Karuna Pande Joshi.Understanding multi-lingual threat intelligence for AI based cyber-defense systems. In
IEEE International Symposium on Technologiesfor Homeland Security , 2018.9] Sagar Samtani, Hongyi Zhu, and Hsinchun Chen. Proactively identifyingemerging hacker threats from the dark web: A diachronic graph embed-ding framework (d-gef).
ACM Transactions on Privacy and Security(TOPS) , 23(4):1–33, 2020.[10] Nolan Arnold, Mohammadreza Ebrahimi, Ning Zhang, Ben Lazarine,Mark Patton, Hsinchun Chen, and Sagar Samtani. Dark-net ecosystemcyber-threat intelligence (cti) tool. In
International Conference onIntelligence and Security Informatics , pages 92–97. IEEE, 2019.[11] Sandeep Narayanan, Ashwini Ganesan, Karuna Joshi, Tim Oates, Anu-pam Joshi, and Tim Finin. Early detection of cybersecurity threats usingcollaborative cognition. In , pages 354–363, 2018.[12] Nitika Khurana, Sudip Mittal, Aritran Piplai, and Anupam Joshi. Pre-venting poisoning attacks on AI based threat intelligence systems. In , pages1–6. IEEE, 2019.[13] Google Threat Analysis Group. New campaign targeting securityresearchers. https://blog.google/threat-analysis-group/new–campaign-targeting-security-researchers/, 2021.[14] Michele Maasberg, Emmanuel Ayaburi, Charles Liu, and Yoris Au. Ex-ploring the propagation of fake cyber news: An experimental approach.In , 2018.[15] Yevgeniy Vorobeychik and Murat Kantarcioglu. Adversarial machinelearning.
Synthesis Lectures on Artificial Intelligence and MachineLearning , 12(3):1–169, 2018.[16] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learningfor networks. In , pages 855–864, 2016.[17] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.Improving language understanding by generative pre-training. Technicalreport, OpenAI, 2018.[18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.Bert: Pre-training of deep bidirectional transformers for language un-derstanding. arXiv:1810.04805 , 2018.[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attentionis all you need. In
Advances in neural information processing systems ,pages 5998–6008, 2017.[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei,and Ilya Sutskever. Language models are unsupervised multitasklearners.
OpenAI blog , 1(8):9, 2019.[21] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek FWong, and Lidia S Chao. Learning deep transformer models for machinetranslation. arXiv:1906.01787 , 2019.[22] Taihua Shao, Yupu Guo, Honghui Chen, and Zepeng Hao. Transformer-based neural network for answer selection in question answering.
IEEEAccess , 7:26146–26156, 2019.[23] Yang Liu and Mirella Lapata. Text summarization with pretrained en-coders. In
Conf. on Empirical Methods in Natural Language Processingand the 9th Int. Joint Conf. on Natural Language Processing , pages3721–3731. ACL, 2019.[24] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, JaredKaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, GirishSastry, and Amanda Askell. Language models are few-shot learners. arXiv:2005.14165 , 2020.[25] OpenAI. Open AI API. https://openai.com/blog/openai-api/, 2021.[26] Jieh-Sheng Lee and Jieh Hsiang. Patent claim generation by fine-tuningOpenAI GPT-2. arXiv:1907.02052 , 2019.[27] Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, andEduard Hovy. Genaug: Data augmentation for finetuning text genera-tors. In
Deep Learning Inside Out (DeeLIO): The First Workshop onKnowledge Extraction and Integration for Deep Learning Architectures ,pages 29–42, 2020.[28] Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni,Antonio Scala, Guido Caldarelli, H Eugene Stanley, and Walter Quat-trociocchi. The spreading of misinformation online.
Proceedings of theNational Academy of Sciences , 113(3):554–559, 2016.[29] Rutvik Vijjali, Prathyush Potluri, Siddharth Kumar, and Sundeep Teki.Two stage transformer model for COVID-19 fake news detection andfact checking. arXiv:2011.13253 , 2020.[30] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, AliFarhadi, Franziska Roesner, and Yejin Choi. Defending against neuralfake news. In
Advances in neural information processing systems , pages9054–9065, 2019. [31] Aditya Pingle, Aritran Piplai, Sudip Mittal, Anupam Joshi, James Holt,and Richard Zak. Relext: Relation extraction using deep learningapproaches for cybersecurity knowledge graph improvement.
IEEE/ACMInternational Conference on Advances in Social Networks Analysis andMining , 2019.[32] Aritran Piplai, Sudip Mittal, Anupam Joshi, Tim Finin, James Holt, andRichard Zak. Creating cybersecurity knowledge graphs from malwareafter action reports.
IEEE Access , 8:211691–211703, 2020.[33] Peng Gao, Xiaoyuan Liu, Edward Choi, Bhavna Soman, ChinmayaMishra, Kate Farris, and Dawn Song. A system for automated open-source threat intelligence gathering and management. arXiv preprintarXiv:2101.07769 , 2021.[34] Jing Liu, Yuan Wang, and Yongjun Wang. The similarity analysis ofmalicious software. In
Int. Conf. on Data Science in Cyberspace . IEEE,2016.[35] Younghee Park, Douglas Reeves, Vikram Mulukutla, and Balaji Sun-daravel. Fast malware classification by automated behavioral graphmatching. In . ACM, 2010.[36] Blake Anderson, Daniel Quist, Joshua Neil, Curtis Storlie, and TerranLane. Graph-based malware detection using dynamic analysis.
Journalin Computer Virology , 7(1):247–258, 2011.[37] Karuna P Joshi, Aditi Gupta, Sudip Mittal, Claudia Pearce, AnupamJoshi, and Tim Finin. Alda: Cognitive assistant for legal documentanalytics. In
AAAI Fall Symposium , 2016.[38] Maithilee Joshi, Sudip Mittal, Karuna P Joshi, and Tim Finin. Semanti-cally rich, oblivious access control using ABAC for secure cloud storage.In
Int. Conf. on edge computing , pages 142–149. IEEE, 2017.[39] Aritran Piplai, Sudip Mittal, Mahmoud Abdelsalam, Maanak Gupta,Anupam Joshi, and Tim Finin. Knowledge enrichment by fusing repre-sentations for malware threat intelligence and behavior. In
InternationalConference on Intelligence and Security Informatics . IEEE, 2020.[40] Aritran Piplai, Priyanka Ranade, Anantaa Kotal, Sudip Mittal, SandeepNarayanan, and Anupam Joshi. Using Knowledge Graphs and Re-inforcement Learning for Malware Analysis. In . IEEE, December 2020.[41] Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and JD Ty-gar.
Adversarial Machine Learning . Cambridge University Press, 2019.[42] MITRE. Virus Total Data Poisoning Case Studies. http://git-hub.com/mitre/advmlthreatmatrix/blob/master/pages/case-studies-page.md
Defence Science Journal , 68(4), 2018.[44] Brian Krebs. Krebs on security. https://krebsonsecurity.com/, 2021.[45] Cyberwire. The CyberWire. https://thecyberwire.com/, 2021.[46] Harold Booth, Doug Rike, and Gregory Witte. The national vulnerabilitydatabase (nvd): Overview. Technical report, National Institute ofStandards and Technology, 2013.[47] aptnotes. APTnotes repository. https://github.com/aptnotes/data, 2021.[48] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layernormalization. stat , 1050:21, 2016.[49] Brian Krebs. Malicious Domain in Solarwinds Hack turned intokillswitch. https://krebsonsecurity.com/2020/12/malicious-domain-in-solarwinds-hack-turned-into-killswitch/, 2021.[50] Peng Gao, Fei Shao, Xiaoyuan Liu, Xusheng Xiao, Haoyuan Liu, ZhengQin, Fengyuan Xu, Prateek Mittal, Sanjeev R Kulkarni, and Dawn Song.A system for efficiently hunting for cyber threats in computer systemsusing threat intelligence. arXiv preprint arXiv:2101.06761