Evaluating Adversarial Evasion Attacks in the Context of Wireless Communications
11 Evaluating Adversarial Evasion Attacks in theContext of Wireless Communications
Bryse Flowers, R. Michael Buehrer, and William C. Headley
Abstract —Recent advancements in radio frequency machinelearning (RFML) have demonstrated the use of raw in-phaseand quadrature (IQ) samples for multiple spectrum sensingtasks. Yet, deep learning techniques have been shown, in otherapplications, to be vulnerable to adversarial machine learning(ML) techniques, which seek to craft small perturbations thatare added to the input to cause a misclassification. The currentwork differentiates the threats that adversarial ML poses toRFML systems based on where the attack is executed from:direct access to classifier input, synchronously transmitted overthe air (OTA), or asynchronously transmitted from a separatedevice. Additionally, the current work develops a methodologyfor evaluating adversarial success in the context of wirelesscommunications, where the primary metric of interest is biterror rate and not human perception, as is the case in imagerecognition. The methodology is demonstrated using the wellknown Fast Gradient Sign Method to evaluate the vulnerabilitiesof raw IQ based Automatic Modulation Classification and con-cludes RFML is vulnerable to adversarial examples, even in OTAattacks. However, RFML domain specific receiver effects, whichwould be encountered in an OTA attack, can present significantimpairments to adversarial evasion.
Index Terms —cognitive radio security, machine learning, mod-ulation classification
I. I
NTRODUCTION
The advent of deep learning has changed the face ofmany fields of research in recent years, including the wire-less communications domain. In particular, Radio FrequencyMachine Learning (RFML), a research thrust championed byDARPA that seeks to develop RF systems that learn fromraw data rather than hand-engineered features, has garneredthe interest of many researchers. One subset of RFML dealswith utilizing raw in-phase and quadrature (IQ) samples forspectrum sensing. Spectrum sensing can be used in DynamicSpectrum Access (DSA) systems to determine the presenceof primary and secondary users in order to adapt transmissionparameters to the environment [1] and has obvious applicationsto signals intelligence. Prior approaches to spectrum sensingwere likelihood or feature based [2]–[7] while more recentapproaches leverage the advances in deep neural networks(DNN) to operate directly on raw IQ samples [8]–[13].While the popularity of RFML has increased, the studyof the vulnerabilities of these systems to adversarial machinelearning [14] has lagged behind. Adversarial machine learning
The work of Bryse Flowers was supported in part by the Bradley MastersFellowship through the Bradley Department of Electrical and ComputerEngineering at Virginia Tech.The authors are with the Bradley Department of Electrical and ComputerEngineering, Virginia Tech, Blacksburg, VA, 24061 USA (e-mail: { brysef,buehrer, cheadley } @vt.edu). consists of learning to apply small perturbations to inputexamples that cause a misclassification. The increased activityin deep learning research in wireless is sure to draw theattention of attackers in this domain but is just beginningto be researched [15], [16]. Adversarial machine learningcould be used, in the context of RFML, to disrupt DSAsystems through primary user emulation [17], evade mobiletransmitter tracking [18], or avoid demodulation by confusingan Automatic Modulation Classification (AMC) system [2].While research thrusts towards adversarial machine learningevasion attacks and defenses can build off of the large bodyof literature present in the Computer Vision (CV) domain,RFML has additional adversarial goals and capabilities beyondthose typically considered in CV. Adversarial goals must besplit between attacks that have direct access to the classifierinput, those that originate from the transmitter and thereforepropagate synchronously with the underlying transmissionthrough a stochastic channel, and those that originate asyn-chronously from a separate transmitter and are only combinedat the receiver or eavesdropper. Additionally, in the contextof wireless communications, attacks must be characterizedagainst the primary metric of interest, bit error rate (BER).An adversary may seek to evade an eavesdropping classifierbut that is of limited benefit if it also corrupts the transmissionto a cooperative receiver.The current work consolidates the additional adversarialgoals and capabilities present in RFML and proposes a newthreat model. Using the well known Fast Gradient SignMethod (FGSM) [19], results are presented from multipleexample attacks against raw-IQ based AMC in order todraw general conclusions about the current vulnerabilities ofRFML systems to adversarial machine learning attacks thathave direct access to the AMC input as well as attacks thatoccur over the air (OTA). The current work is organized asfollows: Section II surveys the related work in this area,Section III presents a consolidated threat model for RFMLsystems, Section IV describes the methodology for executingand evaluating the adversarial evasion attacks in the contextof wireless communications, Section V and VI analyze theattack’s effectiveness with direct access to the classifier inputand in an OTA environment respectively, and conclusions arepresented in Section VII.II. R ELATED W ORK
Threats to machine learning have a wide span in the litera-ture. Causative attacks exert influence over the training processto inject vulnerabilities into the trained classifier [21], [22]. a r X i v : . [ ee ss . SP ] M a r Confidence Reduction UntargetedMisclassification TargetedMisclassification UntargetedMisclassificationTarget Network Architecture and Weights/Bias MatricesOracle (
ADVERSARIAL GOALS A D VE R S A R I A L C A P A B I L I T I E S DECREASING PRIOR KNOWLEDGE INCREASING ATTACK COMPLEXITY
TargetedMisclassificationTarget Network Architecture and Training DatasetSignal Processing
Direct Access Self Protect
Sadeghi & LarssonSectionVSadeghi & Larsson
UntargetedMisclassification TargetedMisclassification
Cover
SectionVI
Fig. 1. Threat Model for RFML signal classification systems presented in the style of [20]. The current work presents results for untargeted misclassificationin both a direct access and self protect scenario with full knowledge of the target network architecture and learned parameters. The related work by Sadeghiand Larsson [15] presented an analysis of two untargeted misclassification attacks against AMC without a channel model applied to the perturbations. Oneattack assumed perfect knowledge of the target network and the other only assumed knowledge of the entire training dataset.
Exploratory attacks [14] seek to learn information about theclassifier. The current work is primarily concerned with eva-sion attacks [23]–[25] which seek to cause a misclassificationat inference time. Specifically, this work uses the well knownFGSM attack, first proposed in [19] for a CV application,as the algorithm for crafting adversarial perturbations due toits low computational complexity; however, the methodologyfor evaluating the attack effectiveness will hold for all currentevasion attacks.Prior security threats to cognitive signal classifiers havebeen researched [26], [27], yet, the state of the art signalclassification systems use deep learning techniques [8]–[13]whose vulnerabilities have not been studied extensively inthe context of RF. In [16] and [28], the authors consideradversarial machine learning for intelligently jamming a deeplearning enabled transmitter, at transmission time and sensingtime, to prevent a transmission. Their work considers learningOTA by observing an acknowledgement from a receiver asa binary feedback. While their work is primarily concernedwith preventing transmission, the current work is primarilyconcerned with enabling transmission while avoiding eaves-droppers and is thus fundamentally different.The work presented in [15] is the closest analogy to thecurrent work. The authors present a study of a similar neuralnetwork architecture [8] using the RML2016.10A dataset [29].The authors present results from attacks on this DNN usingmodifications of FGSM [19] and Universal Adversarial Per-turbations (UAP) [23]. Using their adaptation of UAP, they areable to show black-box results which are time shift invariant,which is a limitation of FGSM. Additionally, the authors usethe energy ratios of the perturbation and modulated signal asan attack constraint, a metric that the current work uses aswell. However, the authors consider perturbations which arebelow the noise floor but implicitly assume they are able to Black-box refers to attacks with full access to the training dataset but noknowledge of the DNN architecture of learned parameter matrices. compromise the eavesdropper’s signal processing chain by notconsidering the effect of the channel on the perturbation signal.Therefore, [15] only considers attacks that have direct-accessto the classifier and aren’t transmitted OTA. The current workexpands upon the study of white-box direct-access attacksagainst RFML systems by exploring the vulnerabilities versusneural network input size. Additionally, the current workconsiders white-box self-protect attacks, which are launchedOTA, where receiver effects can negatively impact adversarialsuccess and must also be evaluated against the effect theperturbation has on the underlying signal transmission bycharacterizing the BER.III. T HREAT M ODEL FOR
RFMLA rich taxonomy already exists for describing threat mod-els for adversarial machine learning in the context of CV;however, threat models which only consider CV applicationslack adversarial goals and capabilities that are unique toRFML. Therefore, the current work extends the threat modelinitially proposed in [20] for RFML in Figure 1. This sectionfirst describes the system model considered for AMC andthen expands on the unique categories of adversarial goalsand capabilities that must be considered when discussingadversarial threats to RFML systems.
A. Automatic Modulation Classification System Model
The current work considers the task of blind signal clas-sification where an eavesdropper attempts to detect a signalin the spectrum, isolate it in time and frequency, and per-form modulation classification. This task assumes that thesignal is a wireless communication between a transmitterand a cooperative receiver where the eavesdropper is notsynchronized and has very limited a priori information about White-box refers to attacks with perfect knowledge of the learned param-eter matrices of the DNN. the communication. Ultimately, the eavesdropper could thenuse the output for DSA, signals intelligence, and/or as apreliminary step to demodulating the signal and extracting theactual information transmitted.The study of adversarial examples in this model could beframed from the perspective of either the eavesdropper or thetransmitter. First, this study can be considered a vulnerabilityanalysis of RFML systems and the information gained can thenbe used to produce a more robust eavesdropper that is hardenedagainst deception by adversarial machine learning. Addition-ally, this study could be considered a feasibility analysisfor methodology to protect transmissions from eavesdroppers.Evading an eavesdropper can limit tracking of the transmitteror automatic demodulation of its transmission. The currentwork does not take a side in the application of this technologyand presents a case for both sides; however, the term adversaryis used to describe the transmitter that seeks to evade aneavesdropper for the remainder of the current work.
B. Adversarial Goals
Three main goals are traditionally considered for adversar-ial machine learning [20]: confidence reduction, untargetedmisclassification, and targeted misclassification. Confidencereduction is the easiest goal an adversary can have. It simplyrefers to introducing uncertainty into the classifier’s decisioneven if it ultimately determines the class of signal correctly.An adversary whose goal is simply to be classified as any othersignal type than its true class, can be described as untargetedmisclassification. Targeted misclassification is typically themost difficult goal of adversarial machine learning. It occurswhen an adversary desires a classifier to output a specific targetclass instead of simply any class that is not the true class. Dueto the hierarchical nature of human engineered modulations,the difficulty of targeted misclassification for AMC dependsheavily on the signal formats of the true and target class.Targeted misclassification are sometimes split between attacksthat start with a real input [19], [25] versus those that startwith noise [30]. The threat model presented in Figure 1 onlyconsiders the former because the current work assumes thatan adversary’s primary goal is to transmit information and notsimply degrade classifier performance.Further, the current work categorizes adversarial goals basedon where the attack is launched from.
1) Direct Access:
Traditional adversarial machine learning,such as those generally considered in CV or the attack consid-ered in [15], fall into the direct access category. This categoryof attack is performed “at the eavesdropper” as part of theirsignal processing chain. Therefore, the propagation channeland receiver effects for the example is known at the time ofcrafting the perturbation, the perturbation is not subjected toany receiver effects, and the perturbation will have no effecton the intended receiver because it is not sent OTA. Attacksat this level are very useful for characterizing the worst casevulnerabilities of a classifier but they are less realistic in thecontext of RFML because it assumes that the signal processingchain has been compromised.
2) Self Protect:
When the adversarial perturbation is addedat the transmitter and propagates along with the transmittedsignal to the eavesdropper, this can be categorized as selfprotect. By adding the perturbation at the transmitter, theperturbation can still be completely synchronous with thesignal transmission; however, the perturbation will now besubjected to all of the receiver effects traditionally consideredin RFML and will also impact the intended receiver. Whilemany of the algorithms that are successful for the directaccess category of attacks will be applicable to self protect,the evaluation of adversarial success must take into accountreceiver effects. Therefore, attacks that seek to create minimalperturbations, such as the modified FGSM method presented in[15], will no longer work because adversarial success can notbe guaranteed due to the signal being subjected to a stochasticprocess.
3) Cover:
RFML allows for a third category of adversarialgoals, in which the adversarial perturbation originates from aseparate emitter from the transmitter and is only combined atthe eavesdropping device. Low cost transmitters can be size,weight, and power (SWaP) constrained. Therefore, it may bebeneficial to have a single unit provide cover for multipleSWaP constrained nodes. However, because these attackscannot rely on synchronization between the transmission andperturbation, the perturbations must be time shift invariant [15]making this category of attack more difficult. The current workdoes not present a study of this category of adversarial goaland leaves that to future work.
C. Adversarial Capabilities
Traditional adversarial machine learning capabilities, suchas those described in [20], generally help with determining“what you want a classifier to see” by providing informa-tion about the target DNN that can subsequently be usedto optimize the input. In the most extreme case, attacksmay have perfect knowledge of the learned parameters ofthe model. These attacks are referred to as white-box. Ina slightly more realistic case, the attacker may have accessto the network architecture and training dataset, but not thelearned parameters. The attacker must then create adversarialexamples that generalize over all possible models created fromthe dataset and architecture. In a very limited case, the attackermay only have access to what is deemed an oracle, an entitythat will label a limited number of
X, Y pairs for the attackerthrough an API [31] or an observable wireless transmission[16], [28]. This allows the attacker to perform limited probesagainst the target network in order to build up an attack.Adversarial machine learning applied to RFML has a dif-ferent class of capabilities an attacker can possess that canbe thought of as “the ability to make a classifier see aspecific example”. RF propagation can be directed throughthe use of smart antennas. Therefore, if a transmitter knewthe location of the receiver, it could direct its energy only atthe receiver, thus minimizing the signal-to-noise ratio (SNR) atthe eavesdropper. Similarly, a jammer could direct energy onlyat the eavesdropper, maximizing the impact of perturbationson classification accuracy while minimizing the impact to thereceiver.
Fig. 2. BPSK adversarial example with a 10 dB ( E s /E j ) perturbation, createdwith the FGSM [19] algorithm, applied. Signal processing chains can present an impediment toadversarial success. Traditionally, RF front ends are builtto reject out of band interference and therefore adversarialperturbations consisting of high frequencies could be filteredout. Power amplifiers can exhibit non-linear characteristicswhich would distort the perturbation. Further, the precisionof the analog to digital converter could limit the attack to stairstepped ranges.
D. Threat Model Assumed in the Current Work
In the current work we assume direct access to the learnedparameters of the target DNN and set the goal as untargetedmisclassification. The current work considers perturbationsthat are specific to the underlying transmitted signal andcharacterizes their effectiveness in the presence of receivereffects such as noise, sample time offsets, and frequencyoffsets. Therefore, both direct access attacks as well as selfprotect are considered. The current work does not assumeknowledge of either the eavesdropper or receiver locations andtherefore does not consider directional antennas and insteadshows results across varying SNR ranges. Further, the currentwork assumes that the receiver is fixed and thus does notintroduce any modifications to the receive chain.IV. M
ETHODOLOGY
Most raw-IQ based signal classifiers seek to take in asignal snapshot, x , and output the most probable class y . Traditionally, x would represent a single channel ofcomplex samples, with little pre-processing performed, andcould therefore be represented as a two-dimensional matrix [ IQ , number of samples ] . Specifically, RFML systems, whichgenerally use DNNs, learn a mapping from the data by solving argmin θ L ( f ( θ , x ) , y )) , (1)where x and y represent the training inputs and target labelsrespectively and f represents the chosen network architecture.A loss function ( L ), such as categorical cross entropy, isgenerally used in conjunction with an optimizer, such asstochastic gradient descent or Adam [32], to train the DNNand thus learn the network parameters θ . While training the model, the dataset is fixed (assuming no data augmentation)and is assumed to be sampled from the same distribution thatwill be seen during operation of the RFML system.Untargeted adversarial machine learning is simply the in-verse of this process. By seeking to maximize the same lossfunction, an adversary can decrease the accuracy of a system.Therefore, the adversary is also solving an optimization prob-lem that can be defined by the following. argmax x ∗ L ( f ( θ , x ∗ ) , y )) (2)In this case, the parameters, θ , of the classifier are fixedbut the input, x ∗ , can be manipulated. Many approachesexist to solve this problem. In particular, FGSM [19] createsuntargeted adversarial examples using x ∗ = x + (cid:15) × sign( ∇ x L ( f ( θ , x ) , y )) , (3)where y represents the true input label and ∇ x representsthe gradient of the loss function with respect to the originalinput, x . This methodology creates adversarial examples con-strained by a distance, (cid:15) , in the feature space in a single step. x ∗ is referred to as an adversarial example. One adversarialexample used in the current work is presented in Figure 2,where the source modulation is BPSK and a perturbation hasbeen applied to achieve untargeted evasion for a direct accessattack.In the context of wireless communications, the absolutevalue of the signal is generally less important than the relativepower of the signal with respect to some other signal such asnoise. Therefore, similar to [15], the current work reformulatesthe perturbation constraint, (cid:15) , from a distance bounding inthe feature space to a bounding of power ratios. Additionally,the signal can be directly evaluated on the primary metric ofinterest, BER, as opposed to the use of human perception, ora proxy for it such as (cid:15) , in CV. Further, A. Adapting FGSM
The average energy per symbol ( E s ) of a transmission canbe computed using E [ E s ] = sps N N (cid:88) i =0 | s i | , (4)where sps represents samples per symbol, N is the totalnumber of samples, and s i represents a particular sample intime. Without loss of generality, the current work assumes theaverage energy per symbol of the modulated signal, E s , is .Therefore, the power ratio of the underlying transmission tothe jamming/perturbation signal ( E j ) can be derived as E s E j = 1 E j = 10 − E j (dB) / (5) Because the perturbation is an electronic signal deliberately crafted toimpair the successful operation of the eavesdropper, the current work usesjamming signal and perturbation signal interchangeably.
PRNG Symbol Encoder RRCInterpolating FIR Filter Time Slice sign X Concatenate + ChannelModelRRCDecimating FIR FilterSymbol DecoderBERCalculateInference Evaluation Time Slice
ModulationDemodulationAMC Evaluation Adversarial ML … … ………
Key bits complex symbolsmodulated signalsliced modulated jamming signalsliced jamming adversarial signaladversarial exampleother ǫ ∇ x Fig. 3. Block diagram of the evaluation methodology developed for the current work. The current work assumes perfect knowledge of the target DNN andtherefore the DNN shown in the AMC Evaluation and Adversarial ML blocks are identical and simply separated for clarity.
Since the input of sign( ∇ x ) in (3) is complex, the outputis also complex, and is therefore a vector whose values are ( ± , ± j ) . Therefore, the magnitude of each sample of thejamming signal can be computed as | sign( ∇ x ) | = | sign( z ) | = (cid:112) ( ± + ( ± = √ (6)Thus the energy per symbol of sign( ∇ x ) can be computedby plugging (6) into (4) resulting in E sign( ∇ x ) = sps N N (cid:88) i =0 | sign( ∇ x ) | = 2 × sps (7)Because sps is fixed throughout transmission, a closed formscaling factor, (cid:15) , can be derived to achieve the desired energyratio ( E s /E j ) by using (cid:15) = (cid:118)(cid:117)(cid:117)(cid:116) E s E j E sign( ∇ x ) = (cid:115) − Ej × sps (8)Plugging (cid:15) into (3) allows the creation of adversarial ex-amples constrained by E s /E j and can be succinctly definedas x ∗ = x + (cid:115) − Ej × sps × sign( ∇ x L ( f ( θ , x ) , y )) (9) Constraining the power ratio in this way can be usefulfor evaluating system design trade-offs. Any given transmitterhas a fixed power budget and the current work considers anadversarial machine learning technique which is not awareof the underlying signal; therefore, power which is usedfor the jamming signal subsequently cannot be used for theunderlying transmission. Future adversarial machine learningtechniques could take into account the bit error rate in theirmethodology which would allow for this energy to accomplishboth purposes, but, this exploration is left to future work. B. Simulation Environment
The high level overview of the simulation environment usedin the current work is shown in Figure 3 and each majorblock is described below. Full evaluation in the context ofwireless communications requires the interfacing of both aDSP and ML framework. The current work uses GNU Radioand PyTorch respectively; however, the methodology is notdependent upon use of those frameworks in any way.
1) Modulation:
The initial modulated signal is generated bya simple flow graph in GNU Radio. Unless otherwise stated,the parameters for transmission can be summarized as follows.The symbol constellations used are BPSK, QPSK, 8PSK, andQAM16. The root raised cosine filter interpolates to samplesper symbol using a filter span of symbols and a roll-off factorof . . examples per modulation scheme are createdusing a random bit stream.
2) Adversarial ML:
In order to craft the jamming signalusing adversarial machine learning techniques it is necessaryto first slice the signal into discrete examples matching theDNN input size. Before feeding these examples into the DNN,dithering is employed to add small amounts of noise to theexamples. The FGSM algorithm is then used to create the perturbations which are concatenated back together to formthe jamming signal. For each E s /E j studied, the jammingsignal is scaled linearly using (8) and added to the modulatedsignal. Unless otherwise stated, E s /E j is swept from to dB with a step size of dB.
3) Channel Model:
The current work considers a simplechannel model with Additive White Gaussian Noise (AWGN)and center frequency offsets. The received signal can becharacterized as follows: S rx ( t ) = e − j πf o t S tx ( t ) + N (0 , σ ) (10)Where f o is the normalized frequency offset and σ isgiven by the desired E s /N . The channel model is againimplemented using a GNU Radio flow graph.
4) Demodulation:
Demodulating the received signal con-sists of match filtering, down-sampling to one sample persymbol, and decoding the symbols back into a bit streamto verify the data received matches the data transmitted. Thedemodulation is also implemented as a GNU Radio flow graphand assumes both symbol and frame synchronization.
5) Automatic Modulation Classification Evaluation:
Top-1accuracy is the metric used for classifier evaluation in [8], [9],and [33] and is the metric we use for evaluation in the currentwork. For untargeted adversarial machine learning, adversarialsuccess is defined as a lower Top-1 accuracy as opposed to ahigher accuracy.
C. Automatic Modulation Classification Target Network1) Network Architecture:
The current work uses the DNNarchitecture first introduced in [8] for raw-IQ modulationclassification. This architecture consists of two convolutionallayers followed by two fully connected layers. This networktakes the IQ samples as a [1 , , N ] tensor which correspondsto 1 channel, IQ, and N input samples. The current workuses extended filter sizes as done in [9] and [33], using filterswith 7 taps and padded with 3 zeros on either side. The firstconvolutional layer has 256 channels, or kernels, and filters Iand Q separately. The first layer does not use a bias term asthis led to vanishing gradients during our training. The secondlayer consists of 80 channels and filters the I and Q samplestogether using a two-dimensional real convolution. This layerincludes a bias term. The feature maps are then flattened andfed into two fully connected layers, the first consisting of 256neurons and the second consisting of the number of outputclasses. All layers use ReLU as the activation function (exceptfor the output layer). As a pre-processing step, the averagepower of each input is normalized to .
2) Dataset A:
The majority of this work uses the opensource RML2016.10A dataset introduced in [29]. This syn-thetic dataset consists of 11 modulation types: BPSK, QPSK,8PSK, CPFSK, GFSK, PAM4, QAM16, QAM64, AM-SSB,AM-DSB, and WBFM. These signals are created inside ofGNU Radio and passed through a dynamic channel model tocreate sample signals at SNRs ranging from -20dB to 18dB.Using an open source dataset allows for quick comparisonof results to those seen in literature; however, this dataset onlyprovides one input size, complex samples. Furthermore,
Fig. 4. Dataset B test accuracy vs SNR for three different neural networkinput sizes. As expected, increasing the input size results in increasing testaccuracy over the entire SNR range studied. this dataset contains limited center frequency offsets. There-fore, it was necessary to create an additional dataset to performall of the evaluations contained in the current work.
3) Dataset B:
This additional dataset was also createdusing synthetic data from GNU Radio. Three datasets werecreated with varying input size (128, 256, and 512). Thesesynthetic datasets consists of modulation schemes: BPSK,QPSK, 8PSK, QAM16, and QAM64. Keeping with theRML2016.10A Dataset, the samples per symbol of the rootraised cosine filter were fixed at . The one sided filter spanin symbols is varied uniformly from 7 to 10 with a step sizeof 1. The roll-off factor of the root raised cosine was varieduniformly from . to . with a step size of . . For thechannel model, the modulated signal was subjected to AWGNand given a center frequency offset as described by (10) tosimulate errors in the receiver’s signal detection stage [33] .The power of the AWGN is calculated using E s /N o and varieduniformly from dB to dB with a step size of . The centerfrequency offset, which was normalized to the sample rate, isswept uniformly from − to with a step size of . .
4) Training Results:
The network is implemented in Py-Torch and trained using an NVIDIA 1080 GPU with theAdam [32] optimizer. The batch size used is 1024 when thenetwork is trained with Dataset A and 512 when trained withDataset B due to the increased example sizes. Models trainedon Dataset A use dropout for regularization, as was initiallyproposed in [8]; however, models trained on Dataset B useBatch Normalization as this increased training stability for thelarger example sizes. For all models, the learning rate is setto . and early stopping is employed with a patience of .During training, 30% of the dataset was withheld as a testset. The remaining 70% of the data is used in the trainingsequence with 5% of the training set used as a validation set.All data is split randomly with the exception that modulationclasses and SNR are kept balanced for all sets. Each of themodels is then evaluated at each SNR in the test set for overallaccuracy and the results are shown in Figure 4. As expected,increasing the input size lead to increasing accuracy. Fig. 5. Overall classification accuracy of a model trained on Dataset A fora direct access attack. This plot compares the classification accuracy whenFGSM in used to apply a specific adversarial perturbation to the accuracywhen “jammed” with a Gaussian noise signal at the same power ratio.
V. A
NALYSIS OF D IRECT A CCESS A TTACKS
In order to first characterize the effectiveness of adversarialmachine learning on raw-IQ based AMC, a baseline study ofaverage classification accuracy against E s /E j was performedusing the model trained on Dataset A. This attack was per-formed with no noise added to the adversarial examples andthus assumes direct access to the classifier input.As can be seen in Figure 5, even at dB, the FGSMattack is more effective than simply adding Gaussian noise(AWGN). At dB, the FGSM attack is effective enoughto degrade the classifier below the performance of randomguessing. This represents an dB improvement over the samedegradation using Gaussian noise.For comparison to other results in CV literature, we can plug E s /E j = 10 dB into (8) which yields that an (cid:15) of ≈ . issufficient for accomplishing the goal of untargeted adversarialmachine learning for direct access attacks on this model. Whilethis clearly shows an improvement over Gaussian jamming,this perturbation is larger than the original example shownin [19] of . for performing an untargeted attack using asource image of a panda. However, that result used ImageNetas a source class and GoogLeNet [34] as the model where theinput dimensions of the image were at least × × ( (cid:29) R , ) while the input size considered here is × × ( R ). Therefore, while we know that the underlyingclassification task is vastly different and exact perturbationconstraints cannot be directly compared, we next investigatewhether increased input dimensionality makes the model moresusceptible to adversarial examples. A. Attack Effectiveness versus NN Input Size
Increasing the DNN input size has been empirically shownto improve the performance of raw-IQ AMC in [33] as wellas the current work’s reproduction of similar results in Figure4. While it is intuitive that viewing longer time windows ofa signal will allow for higher classification accuracy, it isalso intuitive that allowing more adversarial jamming energyto enter the algorithm will have adverse effects. Therefore,the current work presents an experiment used to verify thisintuition. Three copies of the same network, that differ only
Fig. 6. Overall classification accuracy (top) of models trained on Dataset Bin the presence of a direct access FGSM attack. The relative classificationaccuracy ranking of the three different models for each E s /E j (bottom). in input size, are trained on Dataset B. The analysis from theprevious section is then repeated and shown in Figure 6.As expected, at very high E s /E j , where the adversarialenergy is low, the network with the largest input size isthe most accurate. However, it is quickly supplanted by thesecond largest input size when E s /E j drops below dB( (cid:15) ≈ . ). Once E s /E j drops below dB, the classifi-cation accuracy ranking inverts from the initial rankings, withthe smallest input size being the most accurate and the largestinput size being the least accurate. Therefore, when developinga RFML system for use in adversarial environments, thebenefits of increasing input size must be balanced against thecost of increasing the attack surface. B. Analyzing Individual Adversarial Examples
While the earlier subsections presented macro-level results,this subsection presents results at a micro-level by analyzingthe fine grained effect of the adversarial machine learningmethod on individual examples rather than the average effectacross multiple examples. The current work considers a singlemachine learning example from each of the source modula-tions . For each example, E s /E j is swept from to dBwith a step size of a dB. At each E s /E j , the outputs of theDNN before the softmax function (as was shown in [19]) arecaptured.One adversarial example for BPSK is shown in Figure 2. Itcan be seen in the Q samples that, due to the sign operationin (9), the perturbation applied to the signal has a box shape.Therefore, the perturbation alone is easily identifiable; how-ever, in the I samples, where the underlying modulated signalalso lies, it is less distinguishable. Notably, the differencesare most apparent around the symbol locations (note that thissignal has samples per symbol), which could indicate thatthe classifier has learned some notion of synchronization.
1) Difference in Logits:
While the full output of theDNN provides ample information, it is multi-dimensionaland therefore hard to visualize. One metric that is oftenused is a confusion matrix, which captures the relationshipsamong classes. However, confusion matrices are generally While random individual examples are analyzed for simplicity, the con-clusions drawn are further explored in Section VI.
Fig. 7. Output of the model trained on Dataset A for a direct access FGSMattack using a single BPSK adversarial example across varying E s /E j (top)and the corresponding difference in logits (bottom). only presented as an average across multiple examples anddo not provide any notion of the confidence with which aclassifier made the prediction. Therefore, a confusion matrixwould not fully capture the variance of the DNN becausethe outputs would not change unless the input examples weremoved across a decision boundary. Another metric that couldbe used is to apply the softmax function to the output andreport the confidence associated with the source class. Thismetric shows the variance of the classifier output but does notprovide any indication of the Top- accuracy score becauseeven a low confidence output could still be the highest andtherefore the predicted class.The current work presents an additional metric, which weterm the “difference in logits” ( ∆ logits ), that simultaneouslycaptures the accuracy of the classifier as well as the variancein outputs. “Logits” refers to the DNN output before the softmax function has been applied. The maximum output ofall incorrect classes is subtracted from the source (true) classoutput, which can be described by the following Equation. ∆ logits = y s − max( y i ∀ i (cid:54) = s ) (11)The difference in logits can be visualized as the shadedregion in the top of Figures 7 and 8. When ∆ logits is positive,the example is correctly classified and a negative ∆ logits therefore indicates untargeted adversarial success.
2) Classifier Output versus E s /E j : The output of theclassifier for the BPSK example, across multiple E s /E j isshown in Figure 7. At an E s /E j of dB, the jammingintensity present in Figure 2, untargeted misclassification isachieved because the BPSK output is not the highest outputof the classifier; this result is also indicated by viewing that ∆ logits is negative. However, even though misclassification isachieved, the signal is still classified as a linearly modulatedsignal, with the predicted modulation order increasing as E s /E j increased. Linearly modulated signals have symbolswhich exist in the IQ plane (distinguished as solid lines inFigure 7) versus a FSK or continuous signal (distinguished asdashed lines) whose symbols exist in the frequency domainor do not have discrete symbols at all, respectively. Therefore,while the adversarial machine learning method was able to Fig. 8. Output of the model trained on Dataset A for a direct access FGSMattack using a single QAM16 adversarial example across varying E s /E j (top)and the corresponding difference in logits (bottom) achieve untargeted misclassification by causing the classifierto misinterpret the specific linearly modulated signal, theclassifier still captured the hierarchical family of the human-engineered modulation. This reinforces the natural notion thatthe difficulty of targeted adversarial machine learning variesbased on the specific source and target modulations used.Figure 8 shows the output of the classifier for a singleQAM16 example. As was observed in Figure 7, at very low E s /E j , where the attack intensity is the highest, the example isagain classified as QAM (though untargeted misclassificationis narrowly achieved because the model believes it is QAM64).Further, the QAM16 example required much lower energy( E s /E j < dB) than the BPSK example ( E s /E j < dB)to achieve untargeted misclassification. Therefore, increasingthe perturbation energy does not always provide advantageouseffects from the evasion perspective, as can be observed fromthe difference in logits of Figure 8, and the optimal attackintensity varies between source modulations.
3) Mutation Testing with AWGN:
Mutation testing was pro-posed as a defense in [35] where the authors repeatedly applieddomain specific noise to a machine learning example andcalculated the input’s sensitivity, with respect to the classifieroutput, in the presence of this noise. The authors of [35]found that adversarial examples were more sensitive to noisethan examples contained in the initial training distribution andtherefore mutation testing could be used to detect adversarialexamples.The current work presents a study of the effect of AWGN,one of the most prevalent models of noise in RFML, onindividual adversarial examples. For each E s /E j , AWGN isintroduced to the signal at varying E s /N (SNR). E s /N isswept from to dB with a step size of dB. For each ofthe SNRs considered, trials are performed. While E s /E j and E s /N are the parameters swept in this experiment, thejamming to noise ratio ( E j /N ) can be quickly inferred by E j N = E s /N E s /E j = E s N dB − E s E j dB (12) Fig. 9. The effect of noise on the output of the model trained on Dataset Afor a single BPSK adversarial example with an E s /E j of 10 dB. The linerepresents the mean of the difference in logits, at a specific E s /N , whilethe shaded region represents the 25th and 75th percentiles. Again, results are presented in Figure 9 from the BPSKexample originally shown in Figure 2, where E s /E j is dB.The mean of the difference in logits is shown with the 25thand 75th percentiles shaded to show the variance in the outputof the classifier at each SNR. With even a small amount ofnoise ( E s /N of dB) the 75th percentile of the differencein logits becomes positive indicating that the example wasclassified correctly in some iterations. Increasing the noisepower to roughly half that of the applied perturbation ( E j /N of dB) results in the classification, on average, being correct.This effect was not observed across all adversarial examplestested. In Figure 10 it is shown that, while the increasedsensitivity of the classifier output is observed in the samerange of E j /N , it does not result in a correct classification.Therefore, while [35] presented general conclusions that alladversarial examples were sensitive to noise, these resultsshow that this effect is most pronounced when the adversarialperturbation and noise have similar power. Additionally, theseeffects were not observed at all in the individual 8PSK andQAM16 examples studied.This section has shown a baseline result that deep learningbased raw-IQ automatic modulation classification is vulnerableto untargeted adversarial examples. Further, it was shownthat although increasing the neural network input size canimprove accuracy in non-adversarial scenarios, it can makea classifier more susceptible to deception for a given E s /E j .This section also showed that noise can have a negative impacton adversarial success. Therefore, attacks which can onlyprovide a stochastic input to the classifier (self protect) mustbe evaluated differently than attacks that are able to providea deterministic input to the classifier (direct access) and thusthe following section presents a more detailed study of selfprotect attacks using the same adversarial machine learningmethod.VI. A NALYSIS OF S ELF P ROTECT A TTACKS
All OTA attacks must consider the impact of receiver effectson adversarial success; furthermore, self protect attacks mustbalance the secondary goal of evading an adversary with theprimary goal of transmitting information across a wirelesschannel. Neither of these effects have been considered inprior work and therefore, while the previous section studied
Fig. 10. The effect of noise on the output of the model trained on Dataset Afor a single QPSK adversarial example with an E s /E j of 10 dB. The linerepresents the mean of the difference in logits, at a specific E s /N , whilethe shaded region represents the 25th and 75th percentiles. adversarial success in near perfect conditions, this sectionstudies the impact to adversarial success when the examplesare evaluated in the presence of three specific receiver effects,which would likely occur during an OTA attack: AWGN,sample time offsets, and center frequency offsets. A. Additive White Gaussian Noise
AWGN has been shown to negatively impact both BERand classification accuracy. Additionally, as discussed in Sec-tion V-B3, AWGN can have a negative effect on adversarialsuccess. This section further evaluates these negative effectswith a larger scale study. In some cases, such as in “rubbishexamples” [19] or “fooling images” [30], the primary goal ofadversarial machine learning may simply be to create an inputthat is classified with high confidence as some target classstarting from a noise input. However, in general, fooling aclassifier is a secondary goal that must be balanced againstthe primary objective. In CV, this primary objective is topreserve human perception of the image. In the current work,the primary objective of self protect attacks is to transmitinformation to a friendly receiver using a known modulationwhile the secondary objective is to avoid recognition ofthat modulation scheme by an eavesdropper. Therefore, thissection presents results showing the compounding impacts ofadversarial machine learning and AWGN on BER as well asthe effect of AWGN on adversarial success rates.Using the model trained on Dataset A, a range of E s /N and E s /E j are considered. For each E s /N considered, tenthousand trials are executed to provide averaging of therandom variables present in the channel model for a givenrandom signal. The current work considers both the BER andclassification accuracy for BPSK in Figure 11 and 8PSK inFigure 12.Unsurprisingly, increasing the adversarial perturbation en-ergy has positive effects on adversarial success rates (alsoshown previously in Section V) and negative effects on BER.In order to directly compare the trade space between the twoacross a range of SNRs, BER versus classification accuracy isplotted for each E s /E j considered. At high SNR, extremelylow probabilities of bit error, such as those seen in BPSKat E s /N = 20 dB, are hard to characterize empirically.Therefore, in the BER versus classification accuracy plots, Fig. 11. Classification accuracy and bit error rates at varying E s /E j and E s /N for self protect untargeted adversarial attacks using FGSM on themodel trained with Dataset A and a source modulation class of BPSK. all results with lower than − BER have been omitted forclarity.By looking at Figure 11, one can observe that classificationaccuracy can be degraded to ≈ with no noticeable effectto BER for BPSK when using a white-box adversarial attackwith an E s /E j of dB. While this is a very strong result,it only occurs at high SNRs ( > dB). A more reasonableresult to compare to would be the baseline result at dB. Inorder to achieve the same bit error rate as the baseline of noattack (shown as a dashed line), an adversary must increasetheir SNR, and therefore their transmission power, by ≈ dBwhen performing an adversarial attack at an E s /E j of dB.A similar analysis can be performed for QPSK where a dBincrease to SNR is required to maintain the same BER whilereducing classification accuracy to < .As stated in Section V-B, AWGN can have negative affectson adversarial success. Therefore, while an eavesdropper witha high SNR would be fooled nearly all of the time by a BPSKtransmission with an E s /E j of dB, an eavesdropper with an E s /N of dB would still classify this signal correctly of the time. If an adversary wished to attain classificationmore generally for BPSK using FGSM, then they would needto transmit with an E s /E j of dB. This attack intensity wouldrequire an SNR increase of ≈ dB to maintain the same BER.The increased accuracy, at lower SNRs, observed previouslyin Figure 7 can also be observed in Figure 11 and thereforegeneralizes across BPSK examples. This effect can also beobserved, to a lesser extent, in the results of 8PSK (Figure 12).Additional experiments showed that the effect is not observedfor QPSK or QAM16. Note that Figure 10 previously showedthat the increased sensitivity to noise for that QPSK exampledid not result in crossing the decision boundary. The effect ofincreased accuracy cannot be concluded from QAM16 resultsbecause the baseline results already show a slight accuracyimprovement at SNRs around dB.Evaluating attacker success in the case of higher ordermodulations such as 8PSK and QAM16 is less clear. Attacks Fig. 12. Classification accuracy and bit error rates at varying E s /E j and E s /N for self protect untargeted adversarial attacks using FGSM on themodel trained with Dataset A and a source modulation class of 8PSK. with E s /E j ≤ dB already contain bit errors without anyadded noise. Therefore, degrading classification accuracy of8PSK below %, outside of the eavesdropper receiving thesignal at low SNR, would require forward error correction toaccount for the errors in transmission. In the case of QAM16,attacks using E s /E j ≤ dB would impact the receiver morethan the eavesdropper in many scenarios. Specifically, QAM16has a BER of ≈ and ≈ when E s /E j is and dBrespectively even when there is no additive noise. Additionally,note that both of these attack intensities are outside of theoptimal range observed in Figure 8. Therefore, when evaluatedas a function of BER, the classification accuracy is actuallylower in the baseline case than under the presence of thesehigh intensity attacks.In the case of QAM16, lower intensity attacks are effectiveat high SNR; however, they become less effective as SNRdecreases, an anomalous effect previously discussed in SectionV-B. Therefore, untargeted adversarial machine learning withQAM16 as the source modulation class may be most effectivein situations where the eavesdropper is thought to have ahigh fidelity capture of the transmission, such as when theeavesdropper and transmitter are located in close proximity.When the eavesdropper would likely already have a weak viewof the signal, it may be more effective to use physical layersecurity concepts, such as lower transmission power or beamsteering, to further degrade the eavesdropper’s signal capture.These results conclude that adversarial machine learning iseffective across multiple modulations and SNRs to achievethe goal of untargeted misclassification because, for a givenBER, classification can be greatly reduced in many scenarios.However, avoiding signal classification may require sacrificingspectral efficiency or increasing transmission power to main-tain the same bit error rate. Additionally, AWGN was shownto have a negative impact on adversarial success rates in 3out of 4 source modulations tested and therefore adversarialmachine learning can be the most effective at high SNRs. Fig. 13. Classification accuracy vs normalized center frequency offset atvarying E s /E j for self protect untargeted adversarial attacks using FGSM.The model used is trained on Dataset B with an input size of . This datasethas training distribution of ± frequency offset that has been normalizedto the sample rate. B. Frequency Offset
Signal classification systems typically do not know whenand where a transmission will occur. Therefore, they musttake in a wideband signal, detect the frequency bins ofthe signals present, as well as the start and stop times oftransmission, and bring those signals down to baseband forfurther classification. However, this process is not withouterror. One effect shown in [33] was the consequences oferrors in center frequency estimation, resulting in frequencyoffset signals. The authors of [33] found that raw-IQ basedAMC only generalized over the training distribution it wasprovided and therefore if additional frequency offsets outsideof the training distribution were encountered, the classificationaccuracy would suffer. Because these estimations are neverexact, adversarial examples transmitted over the air must alsogeneralize over these effects.In order to evaluate the impact of center frequency offsetsto adversarial examples, it is necessary to use a model thathas been trained to generalize over these effects. Therefore,this experiment uses Dataset B, which has a training distri-bution consisting of ± frequency offsets, which have beennormalized to the sample rate. An input size of is usedfor closer comparison to other results using Dataset A, whichonly has as an input size. The frequency offsets are sweptbetween − . and . with a step size of . . E s /N is evaluated at and dB. At each SNR, 100 trials areperformed to average out the effects of the stochastic process.The results of this experiment are shown in Figure 13.It can be observed that the baseline classifier has learnedto generalize over the effects of frequency offsets within itstraining range of ± ; however, the adversarial examples areclassified with ≈ higher accuracy even at the lowest Fig. 14. Classification accuracy vs time window offsets at varying E s /E j for self protect untargeted adversarial attacks using FGSM. The model usedis trained on Dataset A. evaluated frequency offsets of ± . . This effect is observedat both and dB SNR. Therefore, even minute errors infrequency offset estimation can have negative effects on adver-sarial machine learning and must be considered by adversarialgeneration methods. C. Time Offset
An additional effect that could be encountered is sampletime offsets. In the context of communications, sample timeoffsets can be thought of as a rectangular windowing func-tion, used for creating discrete machine learning examples,not aligning between the adversarial perturbation craftingand signal classification. As previously mentioned, the signalclassification system must estimate the start and stop timesof a transmission; one way to estimate these times is touse an energy detection algorithm where the power of afrequency range is integrated over time and then thresholdedto provide a binary indication of whether a signal is present.A low threshold could have a high false alarm rate and ahigh threshold could induce a lag in the estimation of thestart time. Furthermore, signal classification systems could useoverlapping windows for subsequent classifications to increaseaccuracy through the averaging of multiple classifications ofdifferent “views” of a signal or use non-consecutive windowsdue to real-time computation constraints. Therefore, this effectis a near certainty.This experiment uses the model trained on Dataset A andagain evaluates the effect at an E s /N of and dB.At each SNR, 100 trials are performed. The time offset ismodeled as a shift in the starting index used when slicing thesignal for evaluating the signal classification performance andnon-overlapping/consecutive windows are still used. The timeoffset was swept from 0 to 127 (because the input size is 128and this effect is periodic in the input size); however, only the results from to are shown for simplicity. Time offsetshigher than samples, the symbol period, did not present anysignificant additional impairments beyond those seen at . Theresults are shown in Figure 14.As expected, the network is not heavily effected in thebaseline case. However, the adversarial examples can besignificantly impacted. In the case of an E s /E j of 12dB,simply shifting the time window to the right by four samplescan increase the classification accuracy by . While someadversarial perturbations have been shown to be agnostic tothese time shifts, such as the UAP [23] attack considered in[15], all evaluations of adversarial machine learning in thecontext of RFML, that seek to model OTA attacks, mustassume this effect exists and generalize over it.VII. C ONCLUSIONS AND F UTURE W ORK
The current work has demonstrated the vulnerabilities ofRFML systems to adversarial examples by evaluating multipleexample attacks against a raw-IQ deep learning based modu-lation classifier. First, it was shown that FGSM [19] craftedperturbations were vastly more effective than perturbations thatwere crafted using Gaussian noise at degrading the classifieraccuracy when the attack was launched with direct accessto the classifier input. Furthermore, the current work demon-strated that these vulnerabilities were also present in FGSMbased OTA attacks by evaluating the attack effectiveness inthe presence of three RFML domain specific receiver effects:AWGN, sample time offsets, and center frequency offsets.When evaluating OTA attacks, evading an eavesdropper isgenerally a secondary goal and must be balanced againstthe primary goal of transmission, which is to communicateinformation across a wireless channel. Therefore, the currentwork showed that these attacks harmed the eavesdropper morethan the adversary by demonstrating that, for a given BER,classification accuracy could be lowered for the majority ofthe OTA attacks considered. Given these results, it is logical toconclude that similar vulnerabilities exist in all RFML systemswhen the adversary has white-box knowledge of the classifier.Future RFML systems must consider these vulnerabilitiesand develop defenses against them. The current work hasshown that, while increasing the number of samples usedper classification can increase accuracy in the presence ofAWGN, it can also make the model more susceptible toadversarial examples. Therefore, future RFML systems couldconsider shrinking the input size at the cost of accuracy in thebaseline case. Furthermore, the current work has reinforcedthe viability of mutation testing [35] by showing that RFMLdomain specific receiver effects typically has a negative impacton adversarial examples. Consequently, using classificationsfrom multiple views of the signal, with different sample timeoffsets and center frequency offsets, can aid in detectingadversarial examples and even properly classifying them.However, RFML systems are typically SWaP constrained andtherefore increasing the number of inferences per time stepcould limit the bandwidth that can be sensed in real time.Alternatively, defenses could be incorporated into the DNNtraining phase, which is typically performed offline and thus has more computational resources or no real-time processingconstraint. Ensemble adversarial training [36] has been shownas an effective method for hardening DNN models in theCV domain and the results presented in the current work onBER penalties for adversarial examples can be used to guidewhich examples to include during training. RFML does notnecessarily need to classify all adversarial examples properly,but, it could seek to balance an adversary’s increasing successin evading the eavesdropper versus their degrading ability tocommunicate information.Future OTA adversarial evasion attacks must consider theirability to generalize over RFML domain specific receivereffects as well as their their impact to the underlying transmis-sion. The current work has demonstrated that all three receivereffects considered can degrade the adversary’s ability to evadeclassification. Furthermore, the current work has shown that,while current adversarial methodology can be used for evadingclassification, especially when using a lower order sourcemodulation such as BPSK, it may require sacrificing spectralefficiency or increasing transmission power to maintain thesame bit error rate. Preliminary efforts in presenting additionaladversarial methodology may simply evaluate these effects, aswe have done in this current work. However, more advancedefforts may directly incorporate these models of receivereffects and wireless communications goals directly into theiradversarial methodology in order to create strong adversarialexamples that generalize over receiver effects and have limitedimpact to the underlying transmission.The current work concludes that adversarial machine learn-ing is a credible and evolving threat to RFML systems thatmust be considered in future research.R
EFERENCES[1] S. M. Dudley, W. C. Headley, M. Lichtman, E. Y. Imana, X. Ma,M. Abdelbar, A. Padaki, A. Ullah, M. M. Sohul, T. Yang, and J. H.Reed, “Practical issues for spectrum management with cognitive radios,”
Proc. of the IEEE , vol. 102, no. 3, pp. 242–264, 2014.[2] O. A. Dobre, A. Abdi, Y. Bar-Ness, and W. Su, “Survey of auto-matic modulation classification techniques: classical approaches and newtrends,”
IET Commun. , vol. 1, no. 2, pp. 137–156, 2007.[3] W. C. Headley, J. D. Reed, and C. R. C. M. d. Silva, “Distributed cyclicspectrum feature-based modulation classification,” in
IEEE WirelessCommun. and Netw. Conf. , pp. 1200–1204, 2008.[4] D. T. Kawamoto and R. W. McGwier, “Rigorous moment-based auto-matic modulation classification,”
Proc. of the GNU Radio Conf. , vol. 1,no. 1, 2016.[5] A. Hazza, M. Shoaib, S. A. Alshebeili, and A. Fahad, “An overview offeature-based methods for digital modulation classification,” in , pp. 1–6, 2013.[6] M. M. T. Abdelreheem and M. O. Helmi, “Digital modulation classifica-tion through time and frequency domain features using neural networks,”in , pp. 1–5, 2012.[7] M. Bari, A. Khawar, M. Doroslovaki, and T. C. Clancy, “Recognizingfm, bpsk and 16-qam using supervised and unsupervised learningtechniques,” in ,pp. 160–163, 2015.[8] T. J. OShea, J. Corgan, and T. C. Clancy, “Convolutional radio modu-lation recognition networks,” in
Int. conf. on engineering app. of neuralnetworks , pp. 213–226, Springer, 2016.[9] N. E. West and T. O’Shea, “Deep architectures for modulation recogni-tion,” in
IEEE Int. Symposium on Dynamic Spectrum Access Networks(DySPAN) , pp. 1–6, IEEE, 2017.[10] K. Karra, S. Kuzdeba, and J. Petersen, “Modulation recognition usinghierarchical deep neural networks,” in
IEEE Int. Symposium on DynamicSpectrum Access Networks (DySPAN) , pp. 1–3, 2017. [11] J. L. Ziegler, R. T. Arn, and W. Chambers, “Modulation recognitionwith gnu radio, keras, and hackrf,” in IEEE Int. Symposium on DynamicSpectrum Access Networks (DySPAN) , pp. 1–3, 2017.[12] T. OShea and J. Hoydis, “An introduction to deep learning for thephysical layer,”
IEEE Transactions on Cognitive Commun. and Netw. ,vol. 3, no. 4, pp. 563–575, 2017.[13] L. J. Wong, W. C. Headley, S. Andrews, R. M. Gerdes, and A. J.Michaels, “Clustering learned cnn features from raw i/q data for emitteridentification,” in
IEEE Military Commun. Conf. (MILCOM) , 2018.[14] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. D. Tygar,“Adversarial machine learning,” in
Proc. of the 4th ACM Workshop onSecurity and Artificial Intelligence , AISec ’11, (New York, NY, USA),pp. 43–58, ACM, 2011.[15] M. Sadeghi and E. G. Larsson, “Adversarial attacks on deep-learningbased radio signal classification,”
IEEE Wireless Commun. Letters ,pp. 1–1, 2018.[16] Y. Shi, Y. E. Sagduyu, T. Erpek, K. Davaslioglu, Z. Lu, and J. H. Li,“Adversarial deep learning for cognitive radio security: Jamming attackand defense strategies,” in
IEEE Int. Conf., on Commun. Workshops(ICC Workshops) , pp. 1–6, 2018.[17] R. Chen, J.-M. Park, and J. Reed, “Defense against primary user em-ulation attacks in cognitive radio networks,”
IEEE Journal on SelectedAreas in Commun. , vol. 26, no. 1, pp. 25–37, 2008.[18] K. I. Talbot, P. R. Duley, and M. H. Hyatt, “Specific emitter identificationand verification,”
Technology Review , vol. 113, 2003.[19] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessingadversarial examples,” in
Int. Conf. on Learning Representations , 2015.[20] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, andA. Swami, “The limitations of deep learning in adversarial settings,”in
IEEE European Symposium on Security and Privacy (EuroS&P) ,pp. 372–387, IEEE, 2016.[21] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang,“Trojaning attack on neural networks,” in , The Internet Society, 2018.[22] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotaru, and B. Li,“Manipulating machine learning: Poisoning attacks and countermeasuresfor regression learning,” in
IEEE Symposium on Security and Privacy(SP) , pp. 19–35, 2018.[23] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Univer-sal adversarial perturbations,” , pp. 86–94, 2017.[24] S. Baluja and I. Fischer, “Adversarial transformation networks: Learningto generate adversarial examples,” arXiv:1703.09387 , 2017.[25] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boostingadversarial attacks with momentum,” in
Proc. of the IEEE Conf. onComputer Vision and Pattern Recognition , 2018.[26] T. Newman and T. Clancy, “Security threats to cognitive radio signalclassifiers,” in
Virginia Tech Wireless Personal Commun. Symp. , 2009.[27] T. C. Clancy and A. Khawar, “Security threats to signal classifiers usingself-organizing maps,” in , pp. 1–6, 2009.[28] Y. Shi, T. Erpek, Y. E. Sagduyu, and J. H. Li, “Spectrum data poisoningwith adversarial deep learning.,” in
IEEE Military Commun. Conf.(MILCOM) , 2018.[29] T. J. O’Shea and N. West, “Radio machine learning dataset generationwith gnu radio,” in
Proc. of the GNU Radio Conf. , vol. 1, 2016.[30] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easilyfooled: High confidence predictions for unrecognizable images,” in
Proc.IEEE Conf. on Comp. Vision and Pattern Recog. , pp. 427–436, 2015.[31] N. Papernot, P. D. McDaniel, I. J. Goodfellow, S. Jha, Z. B. Celik,and A. Swami, “Practical black-box attacks against machine learning,”in
Proc. of the 2017 ACM on Asia Conf. on Computer and Commun.Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6,2017 , pp. 506–519, 2017.[32] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
CoRR , vol. abs/1412.6980, 2014.[33] S. C. Hauser, W. C. Headley, and A. J. Michaels, “Signal detectioneffects on deep neural networks utilizing raw iq for modulation classi-fication,” in
Military Commun. Conf. , pp. 121–127, IEEE, 2017.[34] C. Szegedy, L. Wei, J. Yangqing, P. Sermanet, S. Reed, D. Anguelov,D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with con-volutions,” in
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR) , pp. 1–9, 2015.[35] J. Wang, J. Sun, P. Zhang, and X. Wang, “Detecting adversarialsamples for deep neural networks through mutation testing,”
CoRR ,vol. abs/1805.05010, 2018. [36] F. Tram`er, A. Kurakin, N. Papernot, D. Boneh, and P. D. Mc-Daniel, “Ensemble adversarial training: Attacks and defenses,”