Cognitive State Analysis, Understanding, and Decoding from the Perspective of Brain Connectivity
CChapter 1
Cognitive State Analysis,Understanding, and Decodingfrom the Perspective of BrainConnectivity
Junhua Li ∗ , Anastasios Bezerianos, and Nitish Thakor Abstract
Cognitive states are involving in our daily life, which motivates us to explore them andunderstand them by a vast variety of perspectives. Among these perspectives, brain connec-tivity is increasingly receiving attention in recent years. It is the right time to summarizethe past achievements, serving as a cornerstone for the upcoming progress in the field. Inthis chapter, the definition of the cognitive state is first given and the cognitive states thatare frequently investigated are then outlined. The introduction of approaches for estimat-ing connectivity strength is followed. Subsequently, each cognitive state is separately de-scribed and the progress in cognitive state investigation is summarized, including analysis,understanding, and decoding. We concentrate on the literature ascertaining macro-scalerepresentations of cognitive states from the perspective of brain connectivity and give anoverview of achievements related to cognitive states to date, especially within the past tenyears. The discussions and future prospects are stated at the end of the chapter. ∗ Correspondence should be addressed to Junhua Li ([email protected]) a r X i v : . [ q - b i o . N C ] M a y CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY
Cognitive state is defined as the state of a person’s cognitive processes in a dictionary.It reflects the underlying mental action and process related to a wide range of cognitionssuch as memory, attention, evaluation, reasoning, problem-solving, comprehension, andlanguage organization. Therefore, cognitive state assessment endows to probe into cog-nition and be aware of brain state. Questionnaire-based assessment is a quick and easyway to gauge cognitive state with the advantages of low cost and unlimited accessibility.This method is useful and effective to assess quite stable cognitive statuses originated frompathological conditions such as dementia because the cognitive ability is almost not affectedby subjective factors. For instance, the questionnaires of Mini-Mental State Examination(MMSE) [3] and Montreal cognitive assessment (MoCA) [4] are prevalently utilized toassess cognitive capability and to detect cognitive decline. However, questionnaire-basedassessment is not very applicable to measure cognitive states which could vary in a shortperiod. Another drawback of the use of questionnaire-based assessment is the incapabil-ity of a real-time evaluation. It depends on the recall of past engagement and is largelyaffected by subjective factors. In contrast, the assessment based on neurophysiologicalsignals provides an objective evaluation of cognitive states and can perform a real-timeevaluation, but this comes at the expense of high cost and is restricted by the availabilityof measuring equipment and a trained operator. To date, a variety of neurophysiologicalsignals have been being utilized to assess diverse cognitive states. The signals encompasstime series modality (e.g., electroencephalogram (EEG), magnetoencephalogram (MEG))
VIGILANCEEMOTIONWORKLOADFATIGUECognitive StatesSignal Recording etc.DTI fMRIEEG U n i v e r s i t y o f E s s e x U n i v e r s i t y o f E s s e x U n i v e r s i t y o f E s s e x U n i v e r s i t y o f E ss e x U n i v e r s i t y o f E s s e x U n i v e r s i t y o f E s s e x U n i v e r s i t y o f E s s e x . . . Figure 1.1: Overview of analysis and decoding of cognitive states using a variety of signalsincluding time series modality and image modality. Different signal modalities have theirown pros and cons. What modality is adopted depends on the need of an experimentand the purposes of a study. They can be either individually adopted or jointly adopted.This figure incorporates some parts of images from the Google Images with labelling thepermission of reuse and some parts of images from [1, 2] with permissions given by Elsevierand Springer. .1. INTRODUCTION
DTIfMRIEEG Tracts
Eddy CurrentCorrectionMotionCorrectionSlice TimingCorrectionMotion Correctionand RegressionSmothing and FilteringArtifacts Removaland Filtering
BOLD FindingsPerformanceRectified EEGDiverse Signals Signal Processing
PLV PLIWPC CorrelationMI GC OCDC PDCDTFFFDTF SL .. Inter-BrainInter-Region by electrode by anatomical atlas
Connectivity
Figure 1.2: Illustration of the main flowchart for the analysis and classification of cognitivestates. The signals recorded from the human are processed to remove interferences andretain cognition-related information. The processing steps and methods vary for differentmodalities. The critical processing steps for three modalities are delineated in the panelof signal processing. Connective strengths between brains (inter-brain) or between brainregions (inter-region) are estimated by an approach listed in the hexagon to construct aconnectivity network. Subsequently, analysis and classification are performed based on theconnectivity network. The findings revealing brain connectivity representations relatedto cognitive states are presented in the analysis studies while the classification accura-cies are reported in the classification studies. The approaches for connectivity estimationare Phase Locking Value, PLV; Phase Lag Index, PLI; Wavelet Phase Coherence, WPC;Pearson Correlation and Cross Correlation, Correlation; Mutual Information, MI; GrangerCausality, GC; Ordinary Coherence, OC; Directed Coherence, DC; Partial Directed Coher-ence, PDC; Directed Transfer Function, DTF; Full Frequency Directed Transfer Function,FFDTF; Synchronization Likelihood, SL. This figure incorporates some parts of imagesfrom [5, 6] with the reuse permission under an open licence.No matter a study is for analysing and/or decoding cognitive states, a signal measuredfrom a specific location can be considered the proxy of brain activity of that location. Ingeneral, signals are recorded from the human by placing measuring electrodes or sensorsat the targeted locations or scanning a particular body part. The recorded signals shouldbe first processed to remove interferential components (i.e., artifacts). For different signal
CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY modalities, the processing of artifact removal varies. The general and critical processingsteps and methods for three frequently-adopted modalities were shown in the panel of signalprocessing in Fig. 1.2. After the artifact removal, individual regions are separately exploredto investigate cognitive states, which was done by numerous studies. In this way, brainregions are isolated when exploring characteristics relevant to the cognitive state. However,brain regions cooperate together to perform a task while they might dominantly contributeto a particular function required in the task. Therefore, the exploration of interregionalinteractions in the brain is necessary to account for the cooperation. A vast range ofapproaches can be utilized to estimate the interregional interactions in terms of connectivitystrength (see the approaches enclosed in the hexagon in Fig. 1.2). Subsequently, cognitivestates are explored based on the connectivity. To date, researchers have accomplishedsome achievements and made progress in the investigation of cognitive states from theperspective of brain connectivity. We summary these achievements in this chapter to givean overview of the progress. Due to diverse cognitive states and ambiguous boundariesamong cognitive states, we, in this chapter, focus on mental fatigue, mental workload,vigilance, and emotion, which have relatively frequently ascertained and reported in theliterature. The remainder of the chapter is organized as follows. We describe approachesused for estimating connectivity strength in the next section. This is followed by thesections stating the details of achievements in mental fatigue, mental workload, vigilance,and emotion. In section 1.7, toolboxes used for brain connectivity analysis and classificationare introduced. After that, thoughts and further directions are drawn in section 1.8.Finally, a conclusion is given in section 1.9.
Connectivity is a representation of relationships between brain regions or between chan-nels. The extent of interactions between regions or between channels is quantified by themethods estimating connectivity strength and is then represented as a connectivity matrix.Subsequently, the following analysis or classification based on the connectivity matrix (i.e.,comprising connectivity strengths of all pairs of brain regions or channels) can be con-ducted to reveal brain connectivity patterns or to detect cognitive states or brain diseases.The methods frequently used for estimating connectivity strengths are detailed below.
Phase locking value (PLV) shows how two signals measured from two separate brain regionssynchronize in the phase. Let s k ( t ) and s l ( t ) indicate signals measured from brain regions k and l , respectively. The analytical representations of s k ( t ) and s l ( t ) are obtained by theHilbert transform, expressing as [7, 8, 9] z k ( t ) = A k ( t ) e jφ k ( t ) z l ( t ) = A l ( t ) e jφ l ( t ) . (1.1)Phase differences at each time point are then calculated by∆ φ k,l ( t ) = φ k ( t ) − φ l ( t ) . (1.2) .2. METHODS FOR CONNECTIVITY STRENGTH ESTIMATION k and the brain region l is obtained byaveraging over all time points P LV ( k, l ) = 1 n t | n t (cid:88) t =1 e j ∆ φ k,l ( t ) | , (1.3)where n t is the number of time points. PLV is within the range [0 ,
1] with 1 reflecting perfectphase synchronization between brain regions and 0 reflecting no phase synchronization.The PLV calculation is repeated for all pairs of brain regions. After that, all PLVs areassembled to form a connectivity matrix.
When PLV is applied to EEG signal, spurious correlations between EEG signals are intro-duced into connectivity estimation due to that EEG signals from nearby electrodes are verylikely to record the same brain activity source (a.k.a., common source). This is referredto as volume conduction. If two signals measure the same source, their phases should bewell coupled. Therefore, a consistent and non-zero phase lag between signals cannot beattributed to that the signals are from the same source. The method measuring consistentand non-zero phase lag is called phase lag index (PLI) [10]. PLI can be calculated basedon phase differences at each time points ∆ φ k,l ( t ) , t = 1 · · · n t by P LI ( k, l ) = |(cid:104) sign [∆ φ k,l ( t i )] (cid:105)| , (1.4)where (cid:104) · (cid:105) denotes the mean averaged over time, | · | denotes the absolute value, and sign stands for signum function. Similar to the PLV, PLI value ranges from 0 to 1. A valueof 0 reflects either no synchronization or phase synchronization difference centered around0 and π while a value of 1 reflects perfect phase synchronization with consistent phasedifference other than 0 and π [10]. Let φ x ( t, f ) and φ y ( t, f ) be phases at time point t and frequency f for signals x ( t ) and y ( t ), respectively. The phase difference can then be calculated by∆ φ ( t, f ) = φ x ( t, f ) − φ y ( t, f ) (1.5)Wavelet phase coherence is defined in terms of cos ∆ φ ( t, f ) and sin ∆ φ ( t, f ) V ( f ) = (cid:113) (cid:104) cos ∆ φ ( t, f ) (cid:105) + (cid:104) sin ∆ φ ( t, f ) (cid:105) (1.6)where (cid:104) · (cid:105) denotes the mean averaged over time [11]. Pearson correlation is a quantity measuring how a signal x ( t ) correlates with another signal y ( t ), defined as ρ x,y = E [( X − µ x )( Y − µ y )] σ x σ y (1.7) CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY where µ x and µ y are means of signals x ( t ) and y ( t ), respectively, and σ x and σ y are theircorresponding standard deviations.Cross correlation is defined as [12] ρ ( τ ) = (cid:90) ¯ x ( t ) y ( t + τ ) dt (1.8)where ¯ x ( t ) denotes the complex conjugate of x ( t ) and τ is the lag.In the studies with fMRI, Pearson correlation is frequently adopted to estimate func-tional connectivity between voxels or between atlas-based regions or from a seed region toall other regions. Mutual information between two signals is defined as [13] I x,y = H ( x ) + H ( y ) − H ( x, y ) (1.9)where H ( x ) and H ( y ) are Shannon entropies of signals x ( t ) and y ( t ), respectively, and H ( x, y ) is joint entropy. The aforementioned methods estimate connectivity strength for a pair of brain regions ata time. Besides, multivariate autoregressive (MAR) model can be employed to simultane-ously estimate relationships among more than two brain regions. Suppose that there are N signals, representing X = [ x x · · · x N ] T . The current values at each signal aredependent on the past values at each signal, modelling as x ( t ) x ( t )... x N ( t ) = p (cid:88) r =1 A r x ( t − r ) x ( t − r )... x N ( t − r ) + w ( t ) w ( t )... w N ( t ) (1.10)where W = [ w w · · · w N ] T is white uncorrelated noise and p is order of the MARmodel. The model order p can be determined by model quality assessment such as Akaikeinformation criterion (AIC) [14]. The AIC makes a good balance between the goodness offit of the model and the model simplicity. A r , r ∈ { , , · · · , p } are N × N coefficientmatrices for each lag r . A r = a ( r ) a ( r ) · · · · · · a N ( r ) a ( r ) ... ... ... a N ( r )... ... ... a ij ( r ) ...... ... ... ... ... a N ( r ) a N ( r ) · · · · · · a NN ( r ) (1.11) .2. METHODS FOR CONNECTIVITY STRENGTH ESTIMATION a ij ( r ) in the coefficient matrix A r describes linear prediction effect of the r th past value x j ( t − r ) of the signal x j ( t ) on predicting x i ( t ). if any a ij ( r ) (cid:54) = 0, it meansthat x j ( t ) Granger-causes x i ( t ) [15]. It is worth noting that x j ( t ) Granger-causing x i ( t )does not mean that x i ( t ) must Granger-cause x j ( t ). It is not reciprocal. Based on Grangercausality, a few variants have been developed, which are introduced below. MAR model formulated in (1.10) can be rewritten as W ( t ) = p (cid:88) r =0 ¯ A r X ( t − r ) (1.12)where ¯ A r = (cid:26) I, r = 0 − A r , < r ≤ p (1.13)Equation (1.12) can be transformed into frequency domain as X ( f ) = ¯ A − ( f ) W ( f ) (1.14)where ¯ A ( f ) = p (cid:88) r =0 ¯ A r z − r , z = e − πif (1.15)The cross-spectral power density matrix obtained by S ( f ) = E [ X ( f ) X ( f ) H ] = H ( f )Σ H ( f ) H (1.16)where the superscript H represents Hermitian transpose,Σ = σ σ · · · σ N σ σ · · · σ N ... ... . . . ... σ N · · · · · · σ NN (1.17)stands for covariance matrix, and H ( f ) = ¯ A − ( f ) (1.18)Ordinary coherence is defined as [16] C ij ( f ) = | S ij ( f ) | S ii ( f ) S jj ( f ) (1.19)It shows the extent to which brain regions i and j are simultaneously activated. CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY
Directed coherence, unlike ordinary coherence merely describing mutual synchronicity,gives both connective strength and connective direction between brain regions. The di-rected coherence from brain region j to brain region i can be expressed as [17, 18, 19] γ ij ( f ) = σ jj H ij ( f ) (cid:113)(cid:80) Nj =1 σ jj | H ij ( f ) | (1.20) Partial directed coherence (PDC) from brain region j to brain region i is defined as [20, 21] π ij ( f ) = ¯ A ij ( f ) (cid:113)(cid:80) Ni =1 (cid:12)(cid:12) ¯ A ij ( f ) (cid:12)(cid:12) (1.21) Directed transfer function is defined in terms of H as γ ij ( f ) = | H ij ( f ) | (cid:80) Nj =1 | H ij ( f ) | (1.22) γ ij ( f ) quantifies the fraction of inflow to brain region i stemming from brain regions j [19]. After the frequency normalization, directed transfer function is transformed into full fre-quency directed transfer function, expressing as [20] F ij ( f ) = | H ij ( f ) | (cid:80) f (cid:80) Nj =1 | H ij ( f ) | (1.23) Synchronization likelihood quantifies the extent to which a signal recorded from a brainregion synchronized to signals from all the other brain regions. Synchronization likelihoodof signal x ( t ) is defined as [22] S k = 12( w − w t (cid:88) i =1 t (cid:88) j =1 w < | j − i | CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY T a b l e . : M e n t a l F a t i g u e S t ud i e s E x p l o r i n g f r o m t h e P e r s p ec t i v e o f B r a i n C o nn ec t i v i t y R e f e r e n c e C a t e go r y S i g n a l M o d a l - i t y M e t h o dSub j e c t T y p e B o i ss o n e a u l t e t a l.[ a ] [ ] F a t i g u e i np a - t i e n t s w i t h m y a l g i c e n ce ph a l o m y e li - t i s / c h r o n i c f a - t i g u e s y nd r o m e ( M E / C F S ) f M R I S ee d - b a s e d f un c - t i o n a l c o nn ec t i v i t y p a t i e n t s w i t h M E / C F S v s . h e a l t h y c o n t r o l s A n a l y s i s B o i ss o n e a u l t e t a l.[ b ] [ ] F a t i g u e i np a t i e n t s w i t h M E / C F S f M R II nd e p e nd e n t C o m - p o n e n t A n a l y s i s ( I C A ) a ndS ee d - b a s e d f un c t i o n a l c o nn ec t i v i t y p a t i e n t s w i t h M E / C F S v s . h e a l t h y c o n t r o l s A n a l y s i s C r u z G o m ez e t a l.[ ] F a t i g u e i np a t i e n t s w i t h m u l t i p l e s c l e - r o s i s f M R II C A a ndS ee d - b a s e d f un c t i o n a l c o nn ec t i v i t y f a t i g u e dp a t i e n t s v s . n o n - f a t i g u e dp a t i e n t s v s . h e a l t h y c o n t r o l s A n a l y s i s C y n t h i a e t a l.[ ] D r i v i n g f a t i g u e EE G P h a s e L o c k i n g V a l u e ( P L V ) y o un g s ub j ec t s C l a ss i fi c a t i o n C o n t i nu e d o nn e x t p ag e .3. MENTAL FATIGUE T a b l e . c o n t i nu e d f r o m p r e v i o u s p a g e R e f e r e n c e C a t e go r y S i g n a l M o d a l - i t y M e t h o dSub j e c t T y p e C h e n e t a l.[ ] R e a l d r i v i n g e x p e r - i m e n t EE G P h a s e l ag i nd e x y o un g h e a l t h y m a l e s ub - j ec t s A n a l y s i s a nd C l a s - s i fi c a t i o n D i m i t r a k o p o u l o s e t a l.[ ] D r i v i n g f a t i g u e EE G P D C , D T F , a nd P L I h e a l t h y s ub j ec t s C l a ss i fi c a t i o n F i n k ee t a l.[ ] F a t i g u e i np a t i e n t s w i t h m u l t i p l e s c l e - r o s i s f M R I a nd D T I S ee d - b a s e d f un c - t i o n a l c o nn ec t i v i t y m u l t i p l e s c l e r o s i s p a t i e n t s w i t h f a t i g u e v s . h e a l t h y c o n t r o l s A n a l y s i s H a m p s o n e t a l.[ ] F a t i g u e i np a t i e n t s w i t hp e r s i s t e n t b r e a s t c a n ce r f M R II C A a ndS ee d - b a s e d f un c t i o n a l c o nn ec t i v i t y b r e a s t c a n ce r p a t i e n t s w i t h v s . b r e a s t c a n ce r p a t i e n t s w i t h o u t f a t i g u e A n a l y s i s H a r vy e t a l.[ ] D r i v i n g f a t i g u e EE G P a r t i a l d i r ec t e d c o - h e r e n ce ( P D C ) h e a l t h y s ub j ec t s C l a ss i fi c a t i o n H a r vy e t a l.[ ] D r i v i n g f a t i g u e EE G P e a r s o n c o rr e l a t i o n h e a l t h y s ub j ec t s A n a l y s i s K o n g e t a l.[ ] D r i v i n g s i m u l a t i o n EE GG r a n g e r c a u s a li t y h e a l t h y s ub j ec t s A n a l y s i s C o n t i nu e d o nn e x t p ag e CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY T a b l e . c o n t i nu e d f r o m p r e v i o u s p a g e R e f e r e n c e C a t e go r y S i g n a l M o d a l - i t y M e t h o dSub j e c t T y p e K a r e t a l. [ ] P h y s i c a l e x e r c i s e , s i m u l a t e dd r i v i n g , d r i v i n g - r e l a t e d c o m pu t e r i s e d ga m e ( o n l y d r i v i n g - r e l a t e d c o m pu t - e r i s e d ga m e w a s r e p o r t e d ) EE G s y n c h r o n i s a t i o n li k e li h oo d h e a l t h y s ub j ec t s A n a l y s i s L i u e t a l. [ ] M e n t a l f a t i g u e i n - du ce db y c og n i t i v e t a s k EE G D i r ec t e d t r a n s f e r f un c t i o n ( D T F ) h e a l t h y m a l e s ub j ec t s A n a l y s i s N o r d i n e t a l.[ ] P s y c h o m o t o r v i g i - l a n ce t a s k ( P V T ) , m il d t r a u m a t i c b r a i n i n j u r y w i t h p e r s i s t i n g s y m p - t o n s o ff a t i g u e f M R I V o x e l - w i s ec r o ss - c o rr e l a t i o n c o e ffi - c i e n t p a t i e n t s w i t h m il d t r a u - m a t i c b r a i n i n j u r y f a t i g u e v s . h e a l t h y c o n t r o l s A n a l y s i s Q i e t a l. [ ] V i s u a l o ddb a ll t a s k f M R I P e a r s o n c o rr e l a t i o n h e a l t h y s ub j ec t s A n a l y s i s C o n t i nu e d o nn e x t p ag e .3. MENTAL FATIGUE T a b l e . c o n t i nu e d f r o m p r e v i o u s p a g e R e f e r e n c e C a t e go r y S i g n a l M o d a l - i t y M e t h o dSub j e c t T y p e R a m ag e e t a l.[ ] C o n s t a n t e ff o r t t a s k f M R I P e a r s o n c o rr e l a t i o n p a t i e n t s w i t h m il d t r a u - m a t i c b r a i n i n j u r yv s . c o n - t r o l s A n a l y s i s Sun e t a l. [ a ][ ] P V T EE G P D C y o un g s ub j ec t s C l a ss i fi c a t i o n Sun e t a l. [ b ][ ] P V T EE G P D C ( - H z ) s ub j ec t s ( s ub j ec t s e x - c l ud e ddu e t o p oo r m o t i v a - t i o n o n t h e t a s k ) A n a l y s i s Sun e t a l. [ c ][ ] V i s u a l o ddb a ll t a s k f M R I P e a r s o n c o rr e l a t i o n h e a l t h y s ub j ec t s A n a l y s i s W a n g e t a l.[ ] R e a li s t i c D r i v i n g EE G F u zz y s y n c h r o n i z a - t i o n li k e li h oo d h e a l t h y s ub j ec t s ( s ub - j ec t s e x c l ud e ddu e t o p oo r d a t aa c q u i s i t i o n ) A n a l y s i s a nd C l a s - s i fi c a t i o n W a n g e t a l.[ ] D r i v i n g s i m u l a t i o n EE G P h a s e l ag i nd e x h e a l t h y s ub j ec t s A n a l y s i s a nd C l a s - s i fi c a t i o n C o n t i nu e d o nn e x t p ag e CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY T a b l e . c o n t i nu e d f r o m p r e v i o u s p a g e R e f e r e n c e C a t e go r y S i g n a l M o d a l - i t y M e t h o dSub j e c t T y p e X u e t a l. [ ] D r i v i n g w i t h a nd w i t h o u t m e n t a l c a l - c u l a t i o n f N I R S w a v e l e t c o h e r e n ce a nd w a v e l e t ph a s e c o h e r e n ce h e a l t h yy o un g s ub j ec t s A n a l y s i s Z h a n g e t a l.[ ] F a t i g u e i np a t i e n t s w i t h P a r k i n s o n ’ s d i s e a s e ( P D ) f M R I S ee d - b a s e d f un c - t i o n a l c o nn ec t i v i t y P D p a t i e n t s w i t h f a t i g u e v s . P D p a t i e n t s w i t h o u t f a t i g u e v s . h e a l t h y c o n - t r o l s A n a l y s i s Z h ao e t a l.[ ] d r i v i n g f a t i g u e a nd v i s u a l o ddb a ll t a s k EE G O r d i n a r y C o h e r - e n ce ( O C ) h e a l t h yy o un g s ub j ec t s A n a l y s i s .3. MENTAL FATIGUE Most frequently, simulated driving is employed to induce fatigue because the task canbe well defined and be easily controlled in a simulated environment (see Fig. 1.3). Italso has an advantage of safety and participants do not face any risks of physical injury.Responses and task difficulty can be adjusted to meet the requirements of a study. Insuch a driving simulation experiment, participants usually undergo a certain long time ofsimulated driving to attain fatigue state [50, 34, 35, 33, 37]. A much shorter time is enoughwhen highly demanded attention is involved in tasks such as psychomotor vigilance task(PVT) [32, 46]. During the PVT, participants should respond to a counter as quickly asthey can and intense attention is required to perform the task. This imposes high mentalload on the human brain and swiftly depletes brain resource, which facilitates to inducemental fatigue. Another attention-required task that has been used for fatigue studyis visual oddball task [28, 29], in which participants are asked to distinguish the targetletter from shape-analogous English letters by pressing a predefined bottom. Moreover,other attention-based tasks such as detection of three different odd numbers and responsesaccording to image color and arithmetic calculation such as addition and subtraction havealso been utilized for fatigue studies [43]. Besides these laboratory-based experiments, realoutdoor driving lasting about 3.5 hours was conducted to explore mental fatigue [48]. Inaddition, fatigue is investigated not only in healthy people but also in patients. Patientswith fatigue symptom are compared to those who have no such fatigue symptom to revealfatigue effect on diseases (e.g., Parkinson’s disease and multiple sclerosis) [25, 38, 26, 31,27, 32, 45].Figure 1.3: Illustration of driving simulation at an experiment for mental fatigue investi-gation.6 CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY Generally speaking, mental fatigue is related to overall low functional connectivity amongbrain regions. Denser local connectivity (within local communities) and lower long-distanceconnectivity (between communities) are observed in the state of mental fatigue. Thesecontribute to inefficient connectivity for the whole brain network, resulting in low globalefficiency. With relatively dense 64-channel EEG recording, increased clustering coeffi-cient, increased characteristic path length, increased local efficiency, and decreased globalefficiency in topological organization of connectivity network after consecutive implemen-tation of a visual oddball task were found [28], suggesting task implementation towardsmental fatigue led to more active activity within local areas and fewer interactions betweenareas. The regional analysis using betweenness centrality revealed that brain regions lo-cated in the frontal area appeared drastic changes in the connectivity (see Fig. 1.4). Thesimilar changes in clustering coefficient and characteristic path length were also observedin a study with a simulated driving task [30]. These results demonstrated that differenttasks could lead to a similar topological change in brain connectivity network. It is worthmentioning that this is not always the case. In a study using sleep deprivation for inducingfatigue and synchronisation likelihood for estimating brain inter-regional connectivity, thedecreased characteristic path length was observed when a longer time for sleep deprivationwas administered (more fatigued after the longer time for sleep deprivation) [42]. This dis-crepancy might be attributed to different methods used for estimating brain inter-regionalconnectivity and a small number of electrodes (i.e., 19 electrodes) employed for EEG datacollection in the study. With a relatively large cohort of 50 participants, Liu et al. foundthat the coupling among frontal, parietal, and central areas was altered from pre- to post-mental task, and has asymmetric changes in hemisphere [43]. The hemispheric asymmetryof functional connectivity was also observed in the frontoparietal area according to thestudy with PVT task [47]. Using data modality of functional near-infrared spectroscopy(fNIRS), significantly lower wavelet coherence/wavelet phase coherence was observed inthe prefrontal cortex in the frequency bands of 0.6 ∼ ∼ ∼ .3. MENTAL FATIGUE One step closer to practical usage is fatigue classification based on functional connectivity.Fatigue levels can be classified and an appropriate warning is given to drivers or operatorswhose fatigue extent reaches a certain high level. Most of fatigue classification papers aimedto detect driving fatigue in order to eliminate or reduce fatigue-caused traffic accidents[35, 34, 41, 51, 52, 48]. Others are for the purposes of separation of wakefulness and fatiguestates during a task [46]. Connective strengths estimated by the methods presented in theSect. 1.2 (e.g., PLI and PDC [41]) are usually used as feature candidates. Feature selectionis followed to screen feature candidates, retaining discriminative features while excluding8 CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY those features without value in classification. Feature selection can be done by evaluatingindividual features separately [53] or considering a subset of features at a time [41] ordimension reduction algorithms [35]. After that, a classifier named support vector machineis frequently employed to classify fatigue levels based on the selected features [41, 46, 34].Besides, classifiers, linear discriminant analysis, k-nearest neighbours, sequential minimaloptimization, least squares learning, and artificial neural network, have been utilized forfatigue classification [41, 34]. Recently, the fusion of connectivity features and powerfeatures was proposed to classify driving fatigue [35]. The classification performance wasimproved as complementary information was taken in the feature fusion. Mostly, k-foldcross-validation is used to obtain classification accuracy. The procedure of the k-fold cross-validation is as follows:1. Shuffle samples randomly2. Split the samples into k groups (e.g., k = 10)3. Take a group as the testing dataset4. Take the remaining k − k groups to obtain final classification accuracyEach sample is only used in the testing dataset once and used to train the model k − Mental workload is an amount of brain resource paid for task implementation. There hasbeen no clearly universally accepted definition for the mental workload. It is closely re-lated to two determinants: task requirement and human capability or resources [55, 56,57, 58, 59]. Given these two determinants, the mental workload could refer to the por-tion of information processing capacity or resources that is actually required to meet taskdemands [56] or mental workload could be viewed as the ratio of the resources demanded .4. MENTAL WORKLOAD CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY analysis (LDA). Moreover, arithmetic was utilized to induce different workload levels bymodulating addition difficulty. Five workload levels induced by five arithmetical difficultieswere classified by k-nearest neighbours (k-NN) classifier using different feature extractionmethods (i.e., tensor subspace analysis (TSA) and LDA) and multiple bands on spatialdependent representations of phase synchronization. According to the report, a combina-tion of TSA and k-NN using coupling features between the theta in the frontal region andthe upper alpha (10 ∼ 13 Hz) in parieto-occipital region achieved the best performance [63].In the study of using intelligence quotient test for workload induction, multiple featuresincluding connectivity metrics were fed into a neural network to predict workload level [64].In addition, two different categories of mental tasks were used to induce mental workload.As addressed in [65], N-back (0-back and 2-back) and arithmetic (one-digit number additionand three-digit number addition) tasks were used and both within-category classificationand between-category classification were performed. A good performance was obtained forboth classification cases. In a recent study, a regression model was used to establish therelationship between functional connectivity features and workload level in a programmerwho performed code comprehension and syntax error detection [66]. This model can beused to evaluate the programmer’s workload. Besides the tasks with only one participant,workload has been investigated in the case of cooperation between participants. Sciaraffaet al. explored workload among participants who engaged in a collaborative task and foundthat causality value was higher for those who collaborated more tightly [67]. Vigilance is defined as the ability to maintain concentrated attention and to remain alertto stimuli or targets over prolonged periods of time [68, 69]. Vigilance is closely related tofatigue. Fatigue state is accompanied with low vigilance, but the non-fatigue state does nothave to be with high vigilance. It has been found that vigilance was relevant to functionalbrain connectivity [70]. An fMRI study revealed that vigilance was associated with thefluctuations in left amygdala functional connectivity with regions of the salience network[71]. Piantoni et al. used Granger causality to investigate cingulate functional connectivityand found that the forward effective connectivity over the cingulate was related to vigilance[72]. When vigilance is degraded, functional connectivity between brain regions is disrupted[73]. The linkage between functional connectivity and vigilance was also supported by thefinding derived from the analysis of dynamic functional connectivity [74]. Because of therelevance between vigilance and functional connectivity, features derived from functionalconnectivity can be utilised to recognise vigilance. For example, Xie et al. suggested thatthe information derived from the alpha band networks could be used to predict vigilancelevel [75]. It has been shown that phase synchrony indices on connectivity network can beused to successfully predict the average hit response time (a measure indicating vigilance)based on deep neural network model (this is a machine learning method) [76]. An extendedquestion might come up. Is it possible to enhance vigilance when it is declined? To thisend, some efforts have been done. In the stroop color-word task, participants’ vigilance wasenhanced when an auditory stimulus was provided according to behavioural and connec-tivity analysis [77]. The vigilance enhancement could also be attained by visual stimulus[78]. .6. EMOTION Emotion is an affective state of consciousness, which is associated with thoughts, feelings,behavioural responses, and interactions with others [79]. Emotion can be characterized bytwo dimensions of arousal and valence. It is basically divided into six categories: anger,disgust, fear, happiness, sadness and surprise. Images of facial expressions or videos withemotional components (e.g., sounds) are used as stimuli in experiments for emotion studies.According to a study with a large number of participants (i.e., 586), connective strengthwas increased from amygdala to the hippocampus during the encoding of positive andnegative pictures in relation to neutral pictures. The strength of this connection in the re-verse direction showed a smaller elevation [80]. Diseases, such as major depressive disorder(MDD), affect the functional connectivity relevant to emotional stimuli. As indicated inthe study of functional connectivity comparison between healthy group and MDD group,greater functional connectivity between the subgenual anterior cingulate cortex and theamygdala and lower functional connectivity between the subgenual anterior cingulate cor-tex and the insula/putamen, fusiform gyrus, precuneus/posterior cingulate, and middlefrontal gyrus were found in the MDD group [81]. A comparison study showed that func-tional connectivity was obviously different between positive emotion and negative emotion[82]. This suggested that emotion can be recognised based on functional connectivity. Asshown in the emotion recognition study, a classifier of quadratic discriminant analysis wasused to successfully distinguish different emotional states based on connectivity indicessuch as between-channel coherence and correlation [83]. The between-channel connectivestrengths were adopted to distinguish high arousal from low arousal, or positive valencefrom negative valence by traditional machine learning methods (e.g., support vector ma-chine) [84]. Emotion recognition was also achieved by more sophisticated methods suchas graph convolutional neural network [85]. According to recent studies, functional con-nectivity features were better than power spectral density features in emotion recognition[86]. By combining these different kinds of features, emotion recognition performance canbe improved compared to that of using single kind of features [87, 88]. It is not surprisingbecause feature combination could provide more information that single kind of featurescannot provide. In particular, performance improvement is predominant when differentkinds of features are complementary. CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY Based on the progress of understanding of cognitive states and the development of classifi-cation technique for distinguishing cognitive states, it can be seen that neuroimaging datacontain abundant information relevant to cognitive states and brain connectivity analysisusing invasive recorded data is a feasible manner to reveal neural mechanisms underlyingcognitive states. Cognitive state is investigated using experiments, which are designed toinduce required mental state while excluding interferential factors as many as possible.However, absolute exclusion seems impossible and special caution should be given wheninterpreting results. Moreover, artifacts and non-cognitive state background give a con-founding effect on findings. Therefore, preprocessing is a critical step before data analysisand data classification for removing the confounding effect. Until now, many preprocessingmethods have been developed to remove diverse kinds of artifacts such as EOG originatedfrom eye movements and EMG originated from head movements. The principle behindartifacts removal is that raw data are decomposed into components and components corre-sponding to artifacts are removed to eliminate artifacts. Independent component analysis(ICA) [97] principal component analysis (PCA) [98] and wavelet transform [99] belong tothis category. These methods are developed to process data which have been collectedand are not intended to process data in real-time. Nonetheless, some of them have beenmodified to meet the requirement of real-time processing. For example, ICA method was .8. THOUGHTS AND FURTHER DIRECTIONS CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY further from a brain connectivity perspective. Using neurophysiological signals, human in-tentions can be decoded to establish a brain-computer interface for controlling externalassistant devices [108, 109, 110] or interacting with outside world [111, 112, 113]. Theanalogous systems for monitoring cognitive states will be more prevalent than ever. Inorder to facilitate practical use, systems for monitoring cognitive states will be furtherminiaturized to be more portable and more comfortable to users. We have seen this trendin the EEG recording device. Dry electrodes have been adopted in the EEG recording sys-tem to reduce the preparation time before the use. The wireless technique has been appliedfor data transmission so that data cable is not necessary any more, which restrains the mo-bile range of EEG recording. Moreover, smart devices that have been used in daily life willbe more deeply integrated into the monitoring systems of cognitive states. For example,we have developed a driving fatigue monitoring system, where a smartphone is integratedinto the system to show fatigue level in real-time. In the future, appliances will be con-nected to the monitoring systems to better serve human by mutually sharing informationbetween them. For example, the air conditioner automatically adjusts room temperatureaccording to the monitoring data of human states. This can be seen in the near futurebecause all the required techniques are ready. The new generation of communication tech-nique (i.e., 5G) has been being used to largely enhance communication bandwidth, whichmakes it possible to have many things communicated with each other at the same time.By then, comprehensive information including cognitive states is collected from humanand transferred to a clinician for cognitive assessment. As artificial intelligence is rapidlyadvanced, more and more work that has to be done by a clinician before will be taken overby artificial intelligence. With the benefit of super-speed communication technique, heavycomputation demanded by artificial intelligence implementation can be shifted to a remotecomputational server. Such remote access to the computational resource is now available.Amazon web services (AWS) is one of the providers for the service. We believe that theuse of the remote computational resource through the internet will gradually replace sepa-rate local computational workstations. Measured data are uploaded to a remote server forprocessing and analysing. The results are sent back to users for their review and record.With more mechanisms relevant to cognitive states were revealed and the alterationsof cognitive states were more deeply understood, a straightforward question appears inmind: is it possible to delay adverse changes in cognitive states and even to eliminatethe adverse effect? A hint of possibility is given by the study of vigilance enhancementusing visual challenge strategy [114]. Participants were requested to detect intruders ina simulated factory environment under two conditions with and without the presence ofvisual challenge (a rain obscuring the surveillance scene). They found that the visualchallenge results in vigilance enhancement. Another study using a haptic stimulus showedthat the haptic stimulus could help maintain vigilance [115]. As we know, rest can helprecover from fatigue and make the brain refreshed. This viewpoint was supported in termsof brain connectivity according to a study stated that a short break (rest) gave rise toa positive effect on preventing the changes of connectivity pattern towards fatigue [28].These studies are still at very preliminary stage to address the mitigation or preventionof adverse changes of cognitive states, but the feasibility of cognitive modulation at somecases has been shown. In the future, a variety of manners other than those reported inthe above studies will be investigated. This cognitive modulation will be integrated witha cognitive monitoring system so that an unwanted cognitive state could be detected and .8. THOUGHTS AND FURTHER DIRECTIONS aHOFC LOFC HOFCHOFC HOFC LOFC HOFC F3 LOFC LOFCLOFCLOFC P4PzF4Fz P3 LOFC (a) represents correlation (b) (c) LOFCHOFCaHOFCAssociated Topographical Profile AssociatedConnection High-OrderTopographical ProfileTopographicalConnectionTopographicalProfileInter-RegionalConnection Figure 1.6: (a) High-order functional connectivity (HOFC) is obtained by correlation be-tween topographical profiles, which are derived from connectivity strength estimation us-ing any method presented in Sect. 1.2. These connectivity strength estimations representinter-regional connections, which constitute low-order functional connectivity (LOFC). (b)Associated high-order functional connectivity (aHOFC) is a measure assessing similaritybetween the topographical profile and the high-order topographical profile. (c) Illustrationof connections and profiles of the LOFC, HOFC, and aHOFC. Reprinted from [116] withpermission of IEEE.6 CHAPTER 1. COGNITIVE STATE ANALYSIS, UNDERSTANDING, AND DECODING FROM THE PERSPECTIVE OF BRAIN CONNECTIVITY the future, the cross-category classification will be performed. Ideally, a trained classifiercan recognize different levels of diverse cognitive states. Furthermore, these all levels ofcognitive states can be classified in real-time. It is worth noting that low-order functionalconnectivity is currently used for classifying cognitive states. As shown in the latest study,high-order functional connectivity provides additional information characterizing mentalfatigue [5] (see Fig. 1.6 for the illustration. High-order connectivity metrics can be ex-tended as dynamic high-order connectivity metrics by incorporating temporal informationover time.). These features of the high-order functional connectivity can be fused into afeature pool for performance improvement in the classification of cognitive states. In this chapter, we reviewed the researches of cognitive states based on brain connectivity.An overview was first given to depict what was included in this chapter and typical signalprocessing for the analysis and classification of cognitive states. We then described theapproaches which have been utilized to estimate connectivity strength. Subsequently, wesummarised the findings of the data analysis and performances of the classification formental fatigue, mental workload, vigilance, and emotion. Finally, we discussed existingissues and limitations and gave considerations about the future direction in this researchtopic. Overall, mental fatigue was more extensively investigated than the other threecognitive states. Although significant progress was made in the research of cognitive statesbased on brain connectivity, further efforts are required to address the issues and limitationsexisting in the current research of cognitive states. According to the achieved studies, itis undoubted that brain connectivity is one of the promising perspectives to understandcognitive states. ibliography [1] J. Li, L. Zhang: Bilateral adaptation and neurofeedback for brain computer interfacesystem, Journal of neuroscience methods (2), 373–379 (2010)[2] J. Li, A. Cichocki: Deep learning of multifractal attributes from motor imageryinduced EEG, Lecture Notes in Computer Science , 503–510 (2014)[3] M. F. Folstein, S. E. Folstein, P. R. McHugh: Mini-mental state: a practical methodfor grading the cognitive state of patients for the clinician, Journal of psychiatricresearch , 189–198 (1975)[4] Z. S. Nasreddine, N. A. Phillips, V. B´edirian, S. Charbonneau, V. Whitehead,I. Collin, J. L. Cummings, H. Chertkow: The Montreal Cognitive Assessment, MoCA:a brief screening tool for mild cognitive impairment, Journal of the American Geri-atrics Society (4), 695–699 (2005)[5] J. Harvy, N. Thakor, A. Bezerianos, J. Li: Between-Frequency Topographical andDynamic High-Order Functional Connectivity for Driving Drowsiness Assessment,IEEE Transactions on Neural Systems and Rehabilitation Engineering (3), 358–367 (2019)[6] Y. Sun, J. Li, J. Suckling, L. Feng: Asymmetry of hemispheric network topologyreveals dissociable processes between functional and structural brain connectome incommunity-living elders, Frontiers in aging neuroscience , 361 (2017)[7] P. Tass, M. Rosenblum, J. Weule, J. Kurths, A. Pikovsky, J. Volkmann, A. Schnitzler,H.-J. Freund: Detection of n: m phase locking from noisy data: application tomagnetoencephalography, Physical Review Letters (15), 3291 (1998)[8] P. Celka: Statistical analysis of the phase-locking value, IEEE Signal Process. Lett. (9), 577–580 (2007)[9] S. Aydore, D. Pantazis, R. M. Leahy: A note on the phase locking value and itsproperties, Neuroimage , 231–244 (2013)[10] C. J. Stam, G. Nolte, A. Daffertshofer: Phase lag index: Assessment of functionalconnectivity from multi channel EEG and MEG with diminished bias from commonsources, Human Brain Mapping (11), 1178–1193 (2007)278 BIBLIOGRAPHY [11] A. Bandrivskyy, A. Bernjak, P. McClintock, A. Stefanovska: Wavelet phase coherenceanalysis: application to skin temperature and blood flow, Cardiovascular engineering:an international journal (1), 89–93 (2004)[12] R. N. Bracewell, R. N. Bracewell: The Fourier transform and its applications, (1986)[13] T. M. Cover, J. A. Thomas: Elements of information theory, (2012)[14] H. Akaike: A new look at the statistical model identification,, 215–222 (1974)[15] C. W. Granger: Investigating causal relations by econometric models and cross-spectral methods, Econometrica: Journal of the Econometric Society, 424–438 (1969)[16] J. S. Bendat, A. G. Piersol: Random data: analysis and measurement procedures, (2011)[17] S. Schnider, R. Kwong, F. Lenz, H. Kwan: Detection of feedback in the centralnervous system using system identification techniques, Biological cybernetics (3),203–212 (1989)[18] L. Baccala, K. Sameshima, G. Ballester, A. Do Valle, C. Timo-Iaria: Studying theinteraction between brain structures via directed coherence and Granger causality,Applied signal processing (1), 40 (1998)[19] M. J. Kaminski, K. J. Blinowska: A new method of the description of the informationflow in the brain structures, Biological cybernetics (3), 203–210 (1991)[20] M. Kaminski, H. Liang: Causal influence: advances in neurosignal analysis, CriticalReviews in Biomedical Engineering (4), 347–430 (2005)[21] L. A. Baccal´a, K. Sameshima: Partial directed coherence: a new concept in neuralstructure determination, Biological cybernetics (6), 463–474 (2001)[22] C. Stam, B. Van Dijk: Synchronization likelihood: an unbiased measure of gener-alized synchronization in multivariate data sets, Physica D: Nonlinear Phenomena (3-4), 236–251 (2002)[23] F. Takens: Detecting strange attractors in turbulence,, 366–381 (1981)[24] J. Connor, R. Norton, S. Ameratunga, E. Robinson, I. Civil, R. Dunn, J. Bailey,R. Jackson: Driver sleepiness and risk of serious injury to car occupants: populationbased case control study, Bmj (7346), 1125 (2002)[25] J. Boissoneault, J. Letzen, S. Lai, A. O’Shea, J. Craggs, M. E. Robinson, R. Staud:Abnormal resting state functional connectivity in patients with chronic fatigue syn-drome: An arterial spin-labeling fMRI study, Magnetic Resonance Imaging (4),603–608 (2016) IBLIOGRAPHY (3),241–247 (2017)[27] J. P. Hampson, S. M. Zick, T. Khabir, B. D. Wright, R. E. Harris: Altered restingbrain connectivity in persistent cancer related fatigue, NeuroImage: Clinical , 305–313 (2015)[28] J. Li, J. Lim, Y. Chen, K. Wong, N. Thakor, A. Bezerianos, Y. Sun: Mid-task breakimproves global integration of functional connectivity in lower alpha band, Frontiersin Human Neuroscience (2016)[29] Y. Sun, J. Lim, Z. Dai, K. F. Wong, F. Taya, Y. Chen, J. Li, N. Thakor, A. Bezeri-anos: The effects of a mid-task break on the brain connectome in healthy participants:A resting-state functional MRI study, NeuroImage (May 2016), 19–30 (2017)[30] C. Zhao, M. Zhao, Y. Yang, J. Gao, N. Rao, P. Lin: The reorganization of humanbrain networks modulated by driving mental fatigue, IEEE journal of biomedical andhealth informatics (3), 743–755 (2017)[31] C. Finke, J. Schlichting, S. Papazoglou, M. Scheel, A. Freing, C. Soemmer, L. M.Pech, A. Pajkert, C. Pf¨uller, J. T. Wuerfel, C. J. Ploner, F. Paul, A. U. Brandt: Al-tered basal ganglia functional connectivity in multiple sclerosis patients with fatigue,Multiple Sclerosis (7), 925–934 (2015)[32] L. E. Nordin, M. C. M¨oller, P. Julin, A. Bartfai, F. Hashim, T. Q. Li: Post mTBIfatigue is associated with abnormal brain functional connectivity, Scientific Reports (September 2015), 1–12 (2016)[33] W. Kong, W. Lin, F. Babiloni, S. Hu, G. Borghini: Investigating driver fatigueversus alertness using the granger causality network, Sensors (Switzerland) (8),19181–19198 (2015)[34] A. Cynthia, G. Patricia: A new system for detecting fatigue and sleepiness using brainconnectivity EEG based estimation of fatigue, vigilance and sleepiness for drivers,,0–3 (2017)[35] J. Harvy, E. Sigalas, N. Thakor, A. Bezerianos, J. Li: Performance Improvementof Driving Fatigue Identification Based on Power Spectra and Connectivity UsingFeature Level and Decision Level Fusions. In: the 40th International Conference ofthe IEEE Engineering in Medicine and Biology Society ( , Honolulu, USA 2018)pp. 102–105[36] S. I. Dimitriadis, Y. Sun, K. Kwok, N. A. Laskaris, A. Bezerianos: A tensorial ap-proach to access cognitive workload related to mental arithmetic from EEG functionalconnectivity estimates. In: the 35th Annual International Conference of the IEEEEMBS (IEEE, Osaka, Japan 2013) pp. 2940–29430 BIBLIOGRAPHY [37] H. Wang, X. Liu, J. Li, T. Xu, A. Bezerianos, Y. Sun, F. Wan: Driving FatigueRecognition with Functional Connectivity Based on Phase Synchronization, IEEETransactions on Cognitive and Developmental Systems (2020)[38] C. W. Gay, M. E. Robinson, S. Lai, A. O’Shea, J. Craggs, D. D. Price, R. Staud:Abnormal Resting-State Functional Connectivity in Patients with Chronic FatigueSyndrome: Results of Seed and Data-Driven Analyses, Brian Connectivity (1), 48–56 (2016)[39] ´A. J. Cruz G´omez, N. Ventura Campos, A. Belenguer, C. ´Avila, C. Forn: RegionalBrain Atrophy and Functional Connectivity Changes Related to Fatigue in MultipleSclerosis, PLoS ONE (10), 1–8 (2013)[40] J. Chen, H. Wang, Q. Wang, C. Hua: Exploring the fatigue affect-ing electroencephalography based functional brain networks during realdriving in young males, Neuropsychologia (April), 200–211 (2019)doi:10.1016/j.neuropsychologia.2019.04.004[41] G. Dimitrakopoulos, I. Kakkos, A. Vrahatis, K. Sgarbas, J. Li, Y. Sun, A. Bezerianos:Driving Mental Fatigue Classification Based on Brain Functional Connectivity. In: International Conference on Engineering Applications of Neural Networks ( , 2017)pp. 465–474[42] S. Kar, A. Routray, B. P. Nayak: Functional network changes associated with sleepdeprivation and fatigue during simulated driving: Validation using blood biomarkers,Clinical Neurophysiology (5), 966–974 (2011)[43] J. P. Liu, C. Zhang, C. X. Zheng: Estimation of the cortical functional connectivityby directed transfer function during mental fatigue, Applied Ergonomics (1), 114–121 (2010)[44] P. Qi, L. Gao, J. Meng, N. Thakor, A. Bezerianos, Y. Sun: Effects of Rest-Break onMental Fatigue Recovery Determined by a Novel Temporal Brain Network Analysisof Dynamic Functional Connectivity, IEEE Transactions on Neural Systems andRehabilitation Engineering (1), 62–71 (2020) doi:10.1109/TNSRE.2019.2953315[45] A. E. Ramage, D. F. Tate, A. B. New, J. D. Lewis, D. A. Robin: Effort and fatigue-related functional connectivity in mild traumatic brain injury, Frontiers in Neurology (JAN), 1–11 (2019) doi:10.3389/fneur.2018.01165[46] Y. Sun, J. Lim, J. Meng, K. Kwok, N. Thakor, A. Bezerianos: Discriminative Anal-ysis of Brain Functional Connectivity Patterns for Mental Fatigue Classification,Annals of Biomedical Engineering (10), 2084–2094 (2014)[47] Y. Sun, J. Lim, K. Kwok, A. Bezerianos: Functional cortical connectivity analy-sis of mental fatigue unmasks hemispheric asymmetry and changes in small-worldnetworks, Brain and Cognition (1), 220–230 (2014) IBLIOGRAPHY (02), 1550002 (2015)[49] L. Xu, B. Wang, G. Xu, W. Wang, Z. Liu, Z. Li: Functional connectivity analysisusing fNIRS in healthy subjects during prolonged simulated driving, NeuroscienceLetters , 21–28 (2017)[50] R. Bose, H. Wang, A. Dragomir, N. Thakor, A. Bezerianos, J. Li: Regression BasedContinuous Driving Fatigue Estimation: Towards Practical Implementation, IEEETransactions on Cognitive and Developmental Systems (2019)[51] J. He, G. Zhou, H. Wang, E. Sigalas, N. Thakor, A. Bezerianos, J. Li: BoostingTransfer Learning Improves Performance of Driving Drowsiness Classification UsingEEG,, 1–4 (2018)[52] H. Wang, A. Dragomir, N. I. Abbasi, J. Li, N. V. Thakor, A. Bezerianos: A novelreal-time driving fatigue detection system based on wireless dry EEG, Cognitiveneurodynamics, 1–12 (2018)[53] J. Li, Y. Wang, L. Zhang, A. Cichocki, T.-P. Jung: Decoding EEG in cognitive taskswith time-frequency and connectivity masks, IEEE Transactions on Cognitive andDevelopmental Systems (4), 298–308 (2016)[54] J. Li, Y. Sun, Y. Huang, A. Bezerianos, R. Yu: Machine learning technique revealsintrinsic characteristics of schizophrenia: an alternative method, Brain imaging andbehavior, 1–11 (2018)[55] A. Welford: Mental work-load as a function of demand, capacity, strategy and skill,Ergonomics (3), 151–167 (1978)[56] G. F. Wilson, F. T. Eggemeier: Psychophysiological assessment of workload in multi-task environments, Multiple-task performance (1991)[57] B. M. Huey, C. D. Wickens: Workload transition: Implications for individual andteam performance ( , 1993)[58] D. Gopher: Analysis and measurement of mental load, International perspectives onpsychological science , 265–91 (2013)[59] D. Gopher, E. Donchin: Workload: An examination of the concept., (1986)[60] Z. Dai, J. De Souza, J. Lim, P. Ho, Y. Chen, J. Li, N. Thakor, A. Bezerianos,Y. Sun: Eeg cortical connectivity analysis of working memory reveals topologicalreorganization in theta and alpha bands, Frontiers in Human Neuroscience (2017)[61] S. Ren, J. Li, F. Taya, J. DeSouza, N. Thakor, A. Bezerianos: Dynamic FunctionalSegregation and Integration in Human Brain Network during Complex Tasks, IEEETransactions on Neural Systems and Rehabilitation Engineering (6) (2017)2 BIBLIOGRAPHY [62] S. Charbonnier, R. Roy, R. Doleˇzalov´a, A. Campagne, S. Bonnet: Estimation ofWorking Memory Load using EEG Connectivity Measures, Proceedings of the 9thInternational Joint Conference on Biomedical Engineering Systems and Technologies (February), 122–128 (2016)[63] S. I. Dimitriadis, Y. Sun, K. Kwok, N. A. Laskaris, N. Thakor, A. Bezerianos: Cog-nitive Workload Assessment Based on the Tensorial Treatment of EEG Estimates ofCross-Frequency Phase Interactions, Annals of Biomedical Engineering (4), 977–989 (2015)[64] N. Friedman, T. Fekete, K. Gal, O. Shriki: EEG-based prediction of cogni-tive load in intelligence tests, Frontiers in Human Neuroscience (June) (2019)doi:10.3389/fnhum.2019.00191[65] G. Dimitrakopoulos, I. Kakkos, Z. Dai, J. Lim, A. Bezerianos, Y. Sun: Task-independent mental workload classification based upon common multiband EEGcortical connectivity, IEEE Transactions on Neural Systems and Rehabilitation En-gineering, under revision (2017)[66] M. V. Kosti, K. Georgiadis, D. A. Adamos, N. Laskaris, D. Spinellis, L. Angelis:Towards an affordable brain computer interface for the assessment of programmers’mental workload, International Journal of Human Computer Studies (March),52–66 (2018)[67] N. Sciaraffa, G. Borghini, P. Arico, G. Di Flumeri, S. Bonelli, F. Drogoul, A. Vozzi,V. Ronca, A. Bezerianos, N. V. Thakor, F. Babiloni: Toward a cooperation indexbased on EEG-workload causality: Preliminary findings on aerospace-like tasks, Pro-ceedings of the Annual International Conference of the IEEE Engineering in Medicineand Biology Society, EMBS, 4554–4557 (2019) doi:10.1109/EMBC.2019.8856835[68] R. Parasuraman: Vigilance, monitoring, and search., (1986)[69] J. S. Warm, R. Parasuraman, G. Matthews: Vigilance requires hard mental workand is stressful, Human factors (3), 433–441 (2008)[70] J. Teng, J. L. Ong, A. Patanaik, J. Tandi, J. H. Zhou, M. W. Chee, J. Lim:Vigilance declines following sleep deprivation are associated with two previouslyidentified dynamic connectivity states, NeuroImage (January), 382–390 (2019)doi:10.1016/j.neuroimage.2019.07.004[71] B. M. Baczkowski, T. Johnstone, H. Walter, S. Erk, I. M. Veer: Sliding-window anal-ysis tracks fluctuations in amygdala functional connectivity associated with physi-ological arousal and vigilance during fear conditioning, NeuroImage (March),168–178 (2017)[72] G. Piantoni, B. L. P. Cheung, B. D. Van Veen, N. Romeijn, B. A. Riedner, G. Tononi,Y. D. Van Der Werf, E. J. Van Someren: Disrupted directed connectivity along thecingulate cortex determines vigilance after sleep deprivation, NeuroImage , 213–222 (2013) IBLIOGRAPHY , 631–634 (2019)doi:10.1109/NER.2019.8717003[74] C. Wang, J. L. Ong, A. Patanaik, J. Zhou, M. W. L. Chee: Spontaneous eyelid clo-sures link vigilance fluctuation with fMRI dynamic connectivity states, Proceedingsof the National Academy of Sciences (34), 9653–9658 (2016)[75] S. Xie, Y. Li: EEG effective connectivity networks for an attentive task requiringvigilance based on dynamic partial directed coherence, Journal of integrative neuro-science (1), 111–118 (2020) doi:10.31083/j.jin.2020.01.1234[76] M. Torkamani-Azar, S. D. Kanik, S. A. Ali Ahmed, S. Aydin, M. Cetin: Predictionof Response Time and Vigilance Score in a Sustained Attention Task from Pre-trialPhase Synchrony using Deep Neural Networks, Proceedings of the Annual Interna-tional Conference of the IEEE Engineering in Medicine and Biology Society, EMBS,676–679 (2019) doi:10.1109/EMBC.2019.8856291[77] F. Al-Shargie, U. Tariq, O. Hassanin, H. Mir, F. Babiloni, H. Al-Nashash: Brainconnectivity analysis under semantic vigilance and enhanced mental states, BrainSciences (12) (2019) doi:10.3390/brainsci9120363[78] I. P. Bodala, J. Li, N. V. Thakor, H. Al-Nashash: EEG and eye tracking demonstratevigilance enhancement with challenge integration, Frontiers in human neuroscience , 273 (2016)[79] P. E. Ekman, R. J. Davidson: The nature of emotion: Fundamental questions. (Ox-ford University Press, 1994)[80] M. Fastenrath, D. Coynel, K. Spalek, A. Milnik, L. Gschwind, B. Roozendaal,A. Papassotiropoulos, D. J. F. de Quervain: Dynamic Modulation of Amygdala-Hippocampal Connectivity by Emotional Arousal, Journal of Neuroscience (42),13935–13947 (2014)[81] T. C. Ho, G. Yang, J. Wu, P. Cassey, S. D. Brown, N. Hoang, M. Chan, C. G.Connolly, E. Henje-Blom, L. G. Duncan, M. A. Chesney, M. P. Paulus, J. E. Max,R. Patel, A. N. Simmons, T. T. Yang: Functional connectivity of negative emotionalprocessing in adolescent depression, Journal of Affective Disorders (1), 65–74(2014)[82] J. Sorinas, J. C. Fernandez-Troyano, J. M. Ferrandez, E. Fernandez: Cortical Asym-metries and Connectivity Patterns in the Valence Dimension of the Emotional Brain.,International Journal of Neural Systems, 2050021–2050021 (2020)[83] Y.-Y. Lee, S. Hsieh: Classifying Different Emotional States by Means of EEG- BasedFunctional Connectivity Patterns, PLOS One (4), e95415 (2014)4 BIBLIOGRAPHY [84] F. Z. Jahromy, A. Bajoulvand, M. R. Daliri: Statistical algorithms for emotion clas-sification via functional connectivity, Journal of integrative neuroscience (3), 293–297 (2019) doi:10.31083/j.jin.2019.03.601[85] Z. Wang, Y. Tong, X. Heng: Phase-Locking Value Based Graph ConvolutionalNeural Networks for Emotion Recognition, IEEE Access , 93711–93722 (2019)doi:10.1109/ACCESS.2019.2927768[86] X. Wu, W. L. Zheng, B. L. Lu: Identifying Functional Brain Connectivity Patterns forEEG-Based Emotion Recognition, International IEEE/EMBS Conference on NeuralEngineering, NER , 235–238 (2019) doi:10.1109/NER.2019.8717035[87] F. Al-Shargie, U. Tariq, M. Alex, H. Mir, H. Al-Nashash: Emotion Recog-nition Based on Fusion of Local Cortical Activations and Dynamic FunctionalNetworks Connectivity: An EEG Study, IEEE Access , 143550–143562 (2019)doi:10.1109/ACCESS.2019.2944008[88] P. Li, H. Liu, Y. Si, C. Li, F. Li, X. Zhu, X. Huang, Y. Zeng, D. Yao, Y. Zhang, P. Xu:EEG Based Emotion Recognition by Combining Functional Connectivity Networkand Local Activations, IEEE Transactions on Biomedical Engineering (10), 2869–2881 (2019) doi:10.1109/TBME.2019.2897651[89] A. Delorme, S. Makeig: EEGLAB: an open source toolbox for analysis of single-trialEEG dynamics including independent component analysis, Journal of neurosciencemethods (1), 9–21 (2004)[90] S. M. Smith, M. Jenkinson, M. W. Woolrich, C. F. Beckmann, T. E. Behrens,H. Johansen-Berg, P. R. Bannister, M. De Luca, I. Drobnjak, D. E. Flitney, R. K.Niazy: Advances in functional and structural MR image analysis and implementationas FSL, Neuroimage , S208–S219 (2004)[91] M. Rubinov, O. Sporns: Complex network measures of brain connectivity: uses andinterpretations, Neuroimage (3), 1059–1069 (2010)[92] J. Wang, X. Wang, M. Xia, X. Liao, A. Evans, Y. He: GRETNA: a graph theoreticalnetwork analysis toolbox for imaging connectomics, Frontiers in human neuroscience , 386 (2015)[93] M. Xia, J. Wang, Y. He: BrainNet Viewer: a network visualization tool for humanbrain connectomics, PloS one (7), e68910 (2013)[94] M. Hassan, M. Shamas, M. Khalil, W. El Falou, F. Wendling: EEGNET: An opensource tool for analyzing and visualizing M/EEG connectome, PLoS One (9),e0138297 (2015)[95] R. Oostenveld, P. Fries, E. Maris, J.-M. Schoffelen: FieldTrip: open source soft-ware for advanced analysis of MEG, EEG, and invasive electrophysiological data,Computational intelligence and neuroscience , 1 (2011) IBLIOGRAPHY (Dec), 1889–1918 (2005)[97] T.-P. Jung, S. Makeig, C. Humphries, T.-W. Lee, M. J. Mckeown, V. Iragui, T. J.Sejnowski: Removing electroencephalographic artifacts by blind source separation,Psychophysiology (2), 163–178 (2000)[98] T.-P. Jung, C. Humphries, T.-W. Lee, S. Makeig, M. J. McKeown, V. Iragui, T. J.Sejnowski: Removing electroencephalographic artifacts: comparison between ICAand PCA,, 63–72 (1998)[99] V. Krishnaveni, S. Jayaraman, S. Aravind, V. Hariharasudhan, K. Ramadoss: Au-tomatic identification and removal of ocular artifacts from EEG using wavelet trans-form, Measurement science review (4), 45–57 (2006)[100] S.-H. Hsu, T. R. Mullen, T.-P. Jung, G. Cauwenberghs: Real-time adaptive EEGsource separation using online recursive independent component analysis, IEEEtransactions on neural systems and rehabilitation engineering (3), 309–319 (2016)[101] B. He, Z. Liu: Multimodal functional neuroimaging: integrating functional MRI andEEG/MEG, IEEE reviews in biomedical engineering , 23–40 (2008)[102] J. Li, Y. Chen, F. Taya, J. Lim, K. Wong, Y. Sun, A. Bezerianos: A unified canonicalcorrelation analysis-based framework for removing gradient artifact in concurrentEEG/fMRI recording and motion artifact in walking recording from EEG signal,Medical & biological engineering & computing (9), 1669–1681 (2017)[103] P. J. Allen, O. Josephs, R. Turner: A method for removing imaging artifact fromcontinuous EEG recorded during functional MRI, Neuroimage (2), 230–239 (2000)[104] R. Niazy, C. Beckmann, G. Iannetti, J. Brady, S. Smith: Removal of FMRI environ-ment artifacts from EEG data using optimal basis sets, Neuroimage (3), 720–737(2005)[105] J. Li, C. Li, N. Thakor, A. Cichocki, A. Bezerianos: A new perspective of noiseremoval from EEG. In: ( , 2017) pp. 501–504[106] J. Li, G. Chen, P. Thangavel, H. Yu, N. Thakor, A. Bezerianos, Y. Sun: A roboticknee exoskeleton for walking assistance and connectivity topology exploration in EEGsignal,, 1068–1073 (2016)[107] J. Li, N. Thakor, A. Bezerianos: Unilateral Exoskeleton Imposes Significantly Dif-ferent Hemispherical Effect in Parietooccipital Region, but Not in Other Regions,Scientific reports (1), 13470 (2018)[108] J. Li, J. Liang, Q. Zhao, J. Li, K. Hong, L. Zhang: Design of Assistive WheelchairSystem Directly Steered By Human Thoughts, International Journal of Neural Sys-tems (03), 1350013 (2013)6 BIBLIOGRAPHY [109] G. R. M¨uller-Putz, R. Scherer, G. Pfurtscheller, R. Rupp: EEG-based neuroprosthe-sis control: A step towards clinical practice, Neuroscience Letters (1-2), 169–174(2005)[110] G. Pfurtscheller, G. R. M¨uller, J. Pfurtscheller, H. J. Gerner, R. Rupp: ’Thought’ -Control of functional electrical stimulation to restore hand grasp in a patient withtetraplegia, Neuroscience Letters (1), 33–36 (2003)[111] J. Li, Y. Liu, Z. Lu, L. Zhang: A competitive brain computer interface: Multi-personcar racing system, Proceedings of the Annual International Conference of the IEEEEngineering in Medicine and Biology Society, EMBS, 2200–2203 (2013)[112] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, J. R. Wolpaw: BCI2000:A general-purpose brain-computer interface (BCI) system, IEEE Transactions onBiomedical Engineering (6), 1034–1043 (2004)[113] X. Gao, D. Xu, M. Cheng, S. Gao: A BCI-based environmental controller for themotion-disabled, IEEE Transactions on Neural Systems and Rehabilitation Engineer-ing (2), 137–140 (2003)[114] I. P. Bodala, J. Li, N. V. Thakor, H. Al-nashash: EEG and Eye Tracking DemonstrateVigilance Enhancement with Challenge Integration, (June), 273 (2016)[115] N. I. Abbasi, I. P. Bodala, A. Bezerianos, Y. Sun, H. Al-Nashash, N. V. Thakor:Role of multisensory stimuli in vigilance enhancement- a single trial event relatedpotential study, Proceedings of the Annual International Conference of the IEEEEngineering in Medicine and Biology Society, EMBS, 2446–2449 (2017)[116] J. Li, N. Thakor, A. Bezerianos: Brain functional connectivity in unconstrainedwalking with and without an exoskeleton, IEEE Transactions on Neural Systemsand Rehabilitation Engineering (3), 730–739 (2020)[117] V. Badrinarayanan, A. Kendall, R. Cipolla: Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE transactions on pattern analysisand machine intelligence (12), 2481–2495 (2017)[118] K. He, X. Zhang, S. Ren, J. Sun: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition ( ,2016) pp. 770–778[119] A. Karpathy, F.-F. Li: Deep Visual-Semantic Alignments for Generating Image De-scriptions,, 3128–3137 (2015)[120] G. Papandreou, L.-C. Chen, K. P. Murphy, A. L. Yuille: Weakly-and semi-supervisedlearning of a deep convolutional network for semantic image segmentation. In: Pro-ceedings of the IEEE international conference on computer vision ( , 2015) pp. 1742–1750[121] J. Wan, D. Wang, S. C. H. Hoi, P. Wu, J. Zhu, Y. Zhang, J. Li: Deep learning forcontent-based image retrieval: A comprehensive study. In: Proceedings of the 22ndACM international conference on Multimedia ( , 2014) pp. 157–166 IBLIOGRAPHY (9), 1858–1867 (2018)[123] J. Li, Z. Struzik, L. Zhang, A. Cichocki: Feature learning from incomplete EEG withdenoising autoencoder, Neurocomputing , 23–31 (2015)[124] S. Jirayucharoensak, S. Pan-Ngum, P. Israsena: EEG-based emotion recognition us-ing deep learning network with principal component based covariate shift adaptation,TheScientificWorldJournal , 627892 (2014)[125] H. Cecotti, A. Gr¨aser: Convolutional neural networks for P300 detection with ap-plication to brain-computer interfaces, IEEE Transactions on Pattern Analysis andMachine Intelligence33