A Real-time Control Approach for Unmanned Aerial Vehicles using Brain-computer Interface
AA Real-time Control Approach for Unmanned Aerial Vehicles usingBrain-computer Interface*
Ravi M. Vishwanath , Saumya Kumaar and S N Omkar Abstract — Brain-computer interfacing (BCI) is a technologythat is almost four decades old and it was developed solelyfor the purpose of developing and enhancing the impactof neuroprosthetics. However, in the recent times, with thecommercialization of non-invasive electroencephalogram (EEG)headsets, the technology has seen a wide variety of applicationslike home automation, wheelchair control, vehicle steering etc.One of the latest developed applications is the mind-controlledquadrotor unmanned aerial vehicle. These applications, how-ever, do not require a very high-speed response and givesatisfactory results when standard classification methods likeSupport Vector Machine (SVM) and Multi-Layer Perceptron(MLPC). Issues are faced when there is a requirement forhigh-speed control in the case of fixed-wing unmanned aerialvehicles where such methods are rendered unreliable due tothe low speed of classification. Such an application requires thesystem to classify data at high speeds in order to retain the con-trollability of the vehicle. This paper proposes a novel methodof classification which uses a combination of Common SpatialParadigm and Linear Discriminant Analysis that provides animproved classification accuracy in real time. A non-linear SVMbased classification technique has also been discussed. Further,this paper discusses the implementation of the proposed methodon a fixed-wing and VTOL unmanned aerial vehicles.
I. INTRODUCTIONResearch in neurological studies hit a roadblock when theneed arose to understand brain waves. The complexity ofsuch waves could only be studied using advanced compu-tational tools. Brain-Computer Interfaces (BCIs) or Mind-Machine Interfaces (MMI) were born out of the need tocapture and analyze the signals on computers. BCI is an ar-tificial system that incorporates the communication betweenthe brain and an external device or a computer. [1] [2] Inthis system the activity of the brain during a certain actionis tapped into using electroencephalogram (EEG) deviceswhich are sent to central system to be processed and extractthem into control signals, initially devised for rehabilitationto help people regain motor skills that are lost or absentnow promises a new field of research for both medical andEngineering applications. The first research on BCIs tookplace in 1970 at University of California at Los Angeleswhich was aimed at evaluating the ability of BCI to bolsterneuroprosthetics [3][4].BCI gained further traction when it was used to treatlocked-in syndromes and neuromuscular disconnections. Au-ditory, visual and facial muscles often lose the reliabilityand are often exhausted when in frequent or prolonged All authors are with the Indian Institute of Science, Bangalore* This project was funded by the JATP, Defense Development ReseachOrganization (DRDO), Govt. of IndiaCopyrights - 978-1-5386-5323-4/18/$31.00 ©2018 IEEE use [1]. Every thought or action gives rise to a particularelectrical activity. A state of daydreaming or deep meditationemits delta waves (0-3 Hz). Sleeping emits theta waves (3-7Hz). A state of consciousness emits alpha waves (8-12 Hz).Engagement in a particular activity or problem-solving emitsbeta waves (12-38 Hz) [5][12]. An EEG uses this frequencydomain feature to display the users intent in terms of brainwaves onto an external device to monitor and control anexternal device. The very first of BCI systems were the P300spellers which could read the brain signals and read lettersfor communication between the patients who were physicallychallenged [4]. California medical center came up with brain-controlled prosthetic legs with the idea of helping the patientswith spinal cord injuries walk. The system could be usedby the patient to move the prosthetic in real time [5]. Asimilar system was developed in Washington to restore thehands for the people with neural disconnection [6]. A BCI-controlled wheelchair was developed for the patients whocould not use the joystick or the same. Different models ofthe wheelchair were developed depending on the amount ofcontrol left to the users discretion. [7] The scope of researchof BCI was initially confined to only medical applicationsfor detection of different brain states such as alertness,emotion, attention; however, it has now extended to includeengineering and industrial applications as well. In a paperproposed by Bastian Venthur et. al, the concentration levelof an operator in a factory was examined over a period oftime. They prepared a model that would access the responsetime of an operator to the different levels of alert messages.This would serve helpful in avoiding accidents that couldotherwise occur due to low alertness or fatigue [8]. Theintroduction of BCIs in the field of home automation hasbeen explored by Wei Tuck Lee, et al, wherein they haveprepared a virtual home environment in which an individualcan control appliances directly with the mind [9]. Brain-Computer Interface is also found promoting itself into thefield of Aerospace Engineering, there have been successfulattempts made to control a multi-rotor unmanned aerialvehicle (UAV) using BCI. A successful prototype has beendeveloped wherein a quadcopter is made to maneuvererupward, downward, right and left directions depending on thecommand received by the pilots brain signals [10]. In anothersuccessful attempt by Vijayendra et. al. [18], the authorshave demonstrated 95% accuracy in brain-computer interfacebased control of UAVs, with the trade-off between accuracyand real-time implementation. The proposed approach is veryefficient but has limited rate of 25 Hz. Moreover, there hasbeen no attempt so far to control a fixed-wing UAV due a r X i v : . [ c s . R O ] S e p o the complexity of streaming of data at a very high rateand their subsequent classification of the EEG signals thatis requisite for high-speed UAV control. This paper aims tosolve the complexities involved in the implementation of aBCI controlled fixed wing aircraft. The main contributionof this paper is designing the control methodology formaneuvering a delta-winged UAV using Common SpatialParadigm (CSP) and Linear Discriminant Analysis (LDA)of the EEG signals and its implementation. These methods,with data processing rate of around 97 Hertz, provide a rapidclassification platform for establishing a stable control linkto the UAV and also achieve a high level of classificationaccuracy of 85%. II. METHODOLOGY A. Experimental Subjects
There were a total of 14 subjects involved in this ex-perimental study, out of which 5 were female and 9 malesubjects, all aged between 18-24 years. The Selected Subjectsdid not have any prior experience in any BCI/HMI relatedtasks and a written consent of their participation in the studywas submitted to the ethical committee of Indian Institute ofScience.
B. Acquisition Protocol
A protocol was defined and followed for acquisition ofEEG data from the subjects. The protocol dictates four motorimagery task be performed with rest breaks incorporated inbetween each tasks. This ensures that the motor imagerytasks are reinforced and the retention of information in thesignal is sufficient. Fig.1 visualizes the protocol followed fordata acquisition.Fig. 1: Workflow of MI tasksThe Motor Imagery (MI) Acquisition procedure is splitinto four tasks. Task 1 involves visualizing left-hand motionwithout any physical movement. Task 2 is the same as Task1 but is performed with the right-hand. Task 3 requiresthe subject to visualize left-hand movement along with themovement of fingers and elbow. Task 4 replicates Task 3 butis performed with the right-hand. The EEG data acquiredduring these 4 tasks is depicted in Fig. 2. The variations inthe EEG data are visually discernable and the accuracy withwhich the features can be extracted is solely depended onthe algorithm used.Finger and elbow movements are used to increase the num-ber of activations in the sensory-motor cortex. Time intervalsin which the tasks are performed were kept arrhythmic so thatthe amplitude of activations are preserved which otherwisetend to wane out when performing a repetitive action. Fig. 2: MI tasks for Acquisition
C. Subject Training
Focused thought is critical to design a BCI system ofrequisite accuracy as diffused thought process induces noisein the acquired data. The training of the subjects in thisexperiment were performed with the assistance of a cog-nitive suite called the Xavier Control Panel ,which comesbundled with the Emotiv SDK. The subjects train to focuson the movements of a virtual box under minimal sensorydistractions to enable efficient MI task execution.
D. EEG and Brain Data
The brain activity is recorded with the assistance ofa commercially available EEG headset, called EPOC+ byEmotiv Inc.(Fig. 3) which provides 14-channel EEG data.Fig. 3: EPOC+ Headset (14-channel)The EEG bands in the human brain activity are usuallyclassified as : • Delta - Delta band a frequency of 3 Hz or below. Ittends to be the highest in amplitude and the slowestwaves. • Theta - This band has a frequency of 3.5 to 7.5 Hz andis classified as ”slow” activity. • Alpha - This region is localized between 7.5 and 13 Hzand is usually best seen in the posterior regions of thehead on each side. • Beta - Beta activity is ”fast” activity and has frequencyof 14 Hz and greater.
Gamma - A gamma signal is a pattern of neuraloscillation in humans with a frequency between 25 and100 Hz, though 40 Hz is typical.
E. Algorithms
As discussed in the previous sections, in this study wehave used a combination of common spatial paradigm (CSP)for feature extraction and linear discriminant analysis (LDA)for model training. The algorithm definitions and appropriategraphical analysis are discussed in this section.
1) Commom Spatial Paradigm (CSP):
CSP algorithmdisintegrates the signal into additive components so that theyhave maximal variance difference in two windows.EEG signals are usually formalized as : { E n } N ∈ R ch × time (1)The range of time trials varied from 100 to 300 seconds.Now, to apply EEG signals for classification, we musttransform them first. The transformation to a feature vectortakes place as follows : E n ∈ R ch × time (cid:55)−→ x n ∈ R d (2)Major points of concern here are : • Noise reduction has to be done • Frequency band selection for optimal performance • Channel selectionFeature matrix then could be obtained as : X ∈ R d × N (3)The purpose of using a spatial filter like CSP, in thisstudy, is that the signals provided by the algorithm areeasier to classify even with simple methods. The objective ofthis section is to design spatial filters that result in optimalvariances for the classification of two motor imagery signalsrelated to left and right arm movements. The CSP filter (byMuller-Gerking et al. ) is mathematically written as : S = W T E (4)where W ∈ R d × ch is a spatial filter matrix and S ∈ R d × time is the filtered signal matrix. The fundamental of CSP is tomaximize Eq. 5 : tr W T Σ W (5)with subject to Eq. 6 W T ( Σ + Σ ) W = I (6)where, Σ = Exp E n E Tn tr E n E Tn E n ∈ { class } (7) Σ = Exp E n E Tn tr E n E Tn E n ∈ { class } (8)Above equations are solved with the help of generalizedeigen value problem . Initially, we decompose as : Σ + Σ = UDU T (9)where, U is a collection of eigenvectors, and D is a diagnolmatrix of eigenvalues. Next we try to find the value of P = √ D − U T , and : ˆ Σ = P Σ P T (10)ˆ Σ = P Σ P T (11)Now, any orthonomral matrices V satisfy the condition V T ( ˆ Σ + ˆ Σ ) V = I Finally, we disintegrate as :ˆ Σ = V Λ V T (12)where, V is a collection of eigenvectors, and Λ is a diagnolmatrix of eigenvalues. The CSP filter set is obtained as : W = P T V (13)The descriptions would be : W T Σ W = Λ = λ . . . λ ch (14) W T Σ W = I − Λ = − λ . . . 1 − λ ch (15)where λ ≥ λ ≥ · · · ≥ λ ch . Hence, the first CSP filter ω provides maximal variance for 1 st class, and the last filter ω for 2 nd class. The first and last m filters are selected inthe following manner : W csp = ( ω · · · ω m ω ch − m + · · · ω ch ) ∈ R m × ch (16)and the filtered signal mathematically is : s (t) = W Tcsp e (t) = ( s ( t ) · · · s d ( t )) T , (17)i.e., d = 2m .Feature vector x = ( x , x , . . . , x d ) T , is then calculated as: x i = log (cid:18) var [ s i ( t )] ∑ di = var [ s i ( t )] (cid:19) (18)Now that we have our feature space constructed, we woahead by beginning the model training using LDA. . Linear Discriminant Analysis Fisher’s linear discriminat analysis (LDA), a very popularbinary classifier, is based on mean vectors and covariancematrices of patterns of each individual class. Here, weattempt to convert a d -dimensional vector x to a scalar z as : z = w T x (19)Basically, the LDA provides us with an optimal projection w so that z becomes easy to discriminate. The fundamentalof LDA (criterion) is maximizing : J ( w ) = ( m − m ) s + s , (20)where, • m and m are averages for z n ∈ class 1 and z n ∈ class2 respectively • s and s are variances for z n ∈ class 1 and z n ∈ class2 respectivelyThe variables are so defined as : ( m − m ) = ( w T µ − w T µ )( w T µ − w T µ ) T (21) ( s + s ) = w T Σ w + w T Σ w (22)where, • µ and µ are averages for x n ∈ class 1 and x n ∈ class2 respectively • Σ and Σ are variances for x n ∈ class 1 and x n ∈ class2 respectivelyThe cost function J ( w ) , can then be written as : J ( w ) = w T S B ww T S W w (23)where, S B = ( µ − µ )( µ − µ ) T (24) S W = Σ + Σ (25)and then the final solution is given by the following :ˆ w ∝ S − W ( µ − µ ) (26)Finally, an appropriate z threshold is chose for accuratecategorization of any x by :ˆ w T x ≥ z → x ∈ { class } (27)ˆ w T x < z → x ∈ { class } (28)A classification example from our testing is shown below inFig. 4 :The overall workflow of the process is demonstrated bythe flow diagram in Fig. 5 : Fig. 4: Binary Classification using LDAFig. 5: Workflow for EEG Classification G. Support Vector Machine with Non-Linear Kernel
With the LDA-based approach we were abale to achievea classification rate of 89% for a 2-class input system. How-ever, for a multi-class categorization, we cannot use LDA,so we used non-linear SVM classification technique andimplemented it for 4-class inputs. SVM was used primarilybecause it has avery small memory footprint and is almostreal-time when put to test.Suppose we have a dataset X ( i ) , Y ( i ) , i = 1,2, . . . ,m andX ∈ R d , Y ∈ R with the seperating hyperplane defined as W T X + b = W T X + b > i f Y ( i ) = + W T X + b < i f Y ( i ) = − W T X + b ≥ + f or Y ( i ) = + W T X + b ≤ − f or Y ( i ) = − Y ( i ) ( W T X + b ) ≥ f or ∀ i (33)Our task now is to find a hyperplane(W,b) with maximaldistance between itself and the closest data points, whileobeying the mentioned constraints. Mathematically : max ( || W || ) w . r . t . ( W , b ) (34)The trick to designing a SVM is to solve the DUAL ofthe above inequality. It can be proved that : min primal = max ( minL ( W , b , α )) (35)where the primal problem is defined as : min (cid:18) W T W (cid:19) (36)To begin solving our above problem, we construct theLangrangian : L ( W , b , α ) = || W || − m ∑ i = α i [ Y ( i ) ( W T X ( i ) + b ) − ] (37)where, α is a Langragian multiplier with the condition α i ≥
0. We have to minimize it w.r.t. W and b;we set the respectivederivatives equal to zero. Derivative w.r.t. W and b if set tozero: W = m ∑ i = α i Y ( i ) X ( i ) (38) m ∑ i = α i Y ( i ) = L ( W , b , α ) = D ( α ) = m ∑ i = α i − m ∑ i , j = Y ( i ) Y ( j ) α i α j ( X ( i ) ) T ( X ( j ) ) (40)Now the DUAL problem reduces to the eq.: max D ( α ) = m ∑ i = α i − m ∑ i , j = Y ( i ) Y ( j ) α i α j ( X ( i ) X ( j ) ) (41)Solving the above optimization would give us α i . Moreover,the Karush-Kuhn-Tucker condition is satisfied on this solu-tion : α i [ Y ( i ) ( W T X ( i ) ) − ] = f or i = , , . . . , m (42)Now, W and b could be found using respectively : W = m ∑ i = α i Y ( i ) X ( i ) (43) b = − max i : Y ( i ) = − W ∗ T X ( i ) + min i : Y ( i ) = W ∗ T X ( i ) Φ : X → F be a non-linear map form input X to a higher dimensional featurespace, F. So, the inner product (cid:104) X ( i ) , X ( j ) (cid:105) in the higherdimensions is (cid:104) φ ( i ) , φ ( j ) (cid:105) .The easiest way to compute the inner product in the featurespace (higher dimensional space) is by using the KernelFunction. It is defined as : K ( x , z ) = (cid:104) φ ( i ) , φ ( j ) (cid:105) (45)Graphically, the transformationm would look something asshown in Fig. 6.Fig. 6: Non-linearly seperable data (on the left) projectedonto a space where it is linearly seperable (on the right)using a non-linear Kernel Function.So, the optimization and decision functions are renderedrespectively as : max D ( α ) = m ∑ i = α i − m ∑ i , j = Y ( i ) Y ( j ) α i α j K (cid:104) X ( i ) X ( j ) (cid:105) (46) F ( X ) = Sign ( m ∑ i = α i Y ( i ) K ( X ( i ) , X ) + b ) (47)The above equations are solved with appropriate choiceof the kernel function, which in our case, we chose aPolynomial Kernel Function of degree d , defined as: K ( X , Y ) = ( X T Y + ) d (48)Solving the above equations, we get the classification/pre-diction label for each time stamped EEG data.III. HARDWARE INTERFACEIn order to demonstrate the real-time computational ca-pabilities of our BCI, we integrated it with a fixed wingUAV (2-command control) and to a multi-rotor (4-commandcontrol). A. Delta Wing
The testing was done on a prototype first, before deployingit in actual flight. The output from the prediction framework(LDA based 2 class classification) is sent to a microcontrollerunit. An
Arduino (ATmega328P) board is used to control aprototype elevon of a delta configuration made of chloroplast.he model is pre-programmed to do a particular actionwith the help of servo motors depending upon the type ofinput commands received. The fig below (Fig. 7) shows theworking model.Fig. 7: Prototype Demonstration
B. Multi-Rotor
In order to establish a proof of concept , an off-the-shelfquadrotor UAV platform (Fig. 8) compatible with the Pythonprogramming language was used. The algorithm, also writtenin Python, is hosted on a ground station that is in constantduplex communication with the UAV via WiFi. The UAVhost an array of flight instruments such as • front camera • bottom camera (low field-of-view) • • • a magnetometer • barometer • ultrasonic sensor • motherboardFig. 8: AR Parrot 2.0 used for testingThe inner control loop program is embedded onto themotherboard while the outer navigation loop is dictatedby an open-source Python library called python-ardronewhich is hosted on the ground station. Sensor data fusion isachieved by using Extended Kalman Filtering (EKF) methodextensively. The implemented algorithm sends only high-level commands (NLSVM based 4-class classification) to the UAV platform (Fig. 9) so that the inherent stability is notcompromised. Fig. 9: Real-time testingIV. RESULTSThe interface works pretty well in real time [18], at arate of 90 Hz. Individual results of each subject have beentabulated below :TABLE I: Performance EvaluationSubject A LDA A NLSVM T f ocus T max LDA represents the testing accuracy for LDA (2-class), A
NLSVM represents the testing accuracy for Non-Linear SVM (4-class) classifier, T f ocus represents the averagetime taken to focus on a certain motor imagery task and T max is the maximum focus time.*The increased performance in case of subject 10 couldbe attributed to the fact that the subject had been doing
Yoga for some years. Yoga has been known to enhance mentaland physical performance in terms of memory, focus duration(Akhtar et al. [16]) and physical stabililty (Omkar et al. [17]).V. CONCLUSIONSA system was developed which takes EEG (electroen-cephalography) signals as input, modifies the signal forfeature extraction and interfaced with elevon for controllingit wireless connection. The raw EEG data was extracted fromthe brain of the subject using the EMOTIV EPOC+ headset.The raw EEG data is a result of only intuitive thinkingwithout any actual physical movements. The data was thentransformed in a suitable data form to be processed. Furtheremoval of artefacts, unwanted frequencies and irregular datawas done. The processed data was then used for preparingthe model for machine learning using LDA analysis method.Suitable markers to mark the Event Related Potential (ERP)to train the machine for evaluate were added. A twin datasetis applied to the created model, to calculate the misclassi-fication and the error percentage in the same dataset andthe twin dataset respectively. The error percentage is 11%.An offline analysis was done by encoding the signals intocommands for the fixed wing Elevon. The final interfacingwas done with the Elevon and the repeated creation, testing,evaluation, and deployment of the models were done to reachthis accuracy. The BCI is also tested on an off-the-shelfmulti-rotor, with classification accuracy as high as 90%, for4-class based control. This work can further be extendedto control other kinds of UAV and the complexity can beincreased and customized based on the requirement. Themodularity, remote access and control of interfaced Elevonbased on pure brain signals in a BCI system is demonstrated.ACKNOWLEDGEMENTSWe give warm thanks to
Mehvesh Ibrahim , Satya Shree and
Kapil Bharadwaj for providing assistance with the datacollection for the Delta-Wing. We also thank
Priya Rao , Ankita Verma , Akshay Khokkar and
Likith Reddy from NITSringar, for helping collect data and testing the classificationon the multi-rotor. We also extend our regards to
Nava-neethkrishnan B for the helpful discussions.R