The neural dynamics underlying prioritisation of task-relevant information
Tijl Grootswagers, Amanda K. Robinson, Sophia M. Shatek, Thomas A. Carlson
DDOI: https://doi.org/ . / c. O R I G I N A L A R T I C L E
NBDT } N e u r a l d a t a s c i e n c e / a n a l y s i s
The neural dynamics underlying prioritisation oftask-relevant information
Tijl Grootswagers | Amanda K. Robinson | SophiaM. Shatek | Thomas A. Carlson The MARCS Institute for Brain, Behaviourand Development, Western SydneyUniversity, Sydney, Australia School of Psychology, University ofSydney, Australia
Correspondence
Tijl Grootswagers, Western SydneyUniversity, Sydney, NSW, AustraliaEmail:[email protected]
Funding information
T.A.C: Australian Research CouncilDiscovery Projects (DP160101300 andDP200101787); A.K.R: Australian ResearchCouncil Discovery Early Career ResearchAward (DE200101159)
The human brain prioritises relevant sensory information toperform di erent tasks. Enhancement of task-relevant in-formation requires exible allocation of attentional resources,but it is still a mystery how this is operationalised in thebrain. We investigated how attentional mechanisms oper-ate in situations where multiple stimuli are presented in thesame location and at the same time. In two experiments,participants performed a challenging two-back task on dif-ferent types of visual stimuli that were presented simultane-ously and superimposed over each other. Using electroen-cephalography and multivariate decoding, we analysed thee ect of attention on the neural responses to each individ-ual stimulus. Whole brain neural responses contained con-siderable information about both the attended and unat-tended stimuli, even though they were presented simulta-neously and represented in overlapping receptive elds. Asexpected, attention increased the decodability of stimulus-related information contained in the neural responses, butthis e ect was evident earlier for stimuli that were presentedat smaller sizes. Our results show that early neural responsesto stimuli in fast-changing displays contain remarkable in- formation about the sensory environment but are also mod-ulated by attention in a manner dependent on perceptualcharacteristics of the relevant stimuli. Stimuli, code, anddata for this study can be found at https://osf.io/ zhwp/ . K E Y W O R D S
MVPA, attention, visual perception, Electroencephalography | INTRODUCTION
To e ciently perform a task, our brains continuously prioritise and select relevant information from a constant streamof sensory input. All sensory input is automatically and unconsciously processed, but the depth of processing variesdepending on the task and input characteristics (Grootswagers et al., 2019a; King et al., 2016; Mohsenzadeh et al.,2018; Robinson et al., 2019; Rossion et al., 2015; Rousselet et al., 2002). At what stage in the response is task-relevantinformation prioritised? Neurophysiological methods such as electroencephalography (EEG) and magnetoencephalog-raphy (MEG) have o ered insight into the time-scales at which selective attention operates in the human brain (God-dard et al., 2019). For example, a stimulus that is presented in an attended location evokes a stronger neural responsearound 100ms (e.g., Mangun, 1995; Mangun et al., 1993). Similarly, when a certain feature of a stimulus is attended,the neural correlate of this feature is enhanced (Martinez-Trujillo and Treue, 2004; Maunsell and Treue, 2006), withe ects for basic features (e.g., colour) starting as early as 100ms (e.g., Zhang and Luck, 2009). In a sequence of stimuli,temporal selection of task-relevant target stimuli is reported around 270ms (Kranczioch et al., 2005, 2003; Marti andDehaene, 2017; Sergent et al., 2005; Tang et al., 2020; Wyart et al., 2015). A question that has received considerablyless focus is how these mechanisms interact in situations where multiple stimuli are presented in the same locationand at the same time. Determining the stages of processing a ected by attention in these situations is important forunderstanding selective attention as a whole, and for constructing an overarching theory of attention.Studying neural responses to simultaneously presented stimuli is di cult, as the stimulus-speci c signals areoverlapping. One solution is to display stimuli at di erent presentation rates and analyse neural responses in thematching frequency bands (e.g., Ding et al., 2006; Müller et al., 2006), but this approach does not allow studyingthe underlying temporal dynamics. Another approach is to use multivariate decoding methods, which have recentlyprovided new opportunities to study attentional e ects on information at the individual stimulus level (e.g., Alilovi et al., 2019; Goddard et al., 2019; Marti and Dehaene, 2017; Smout et al., 2019). These methods also allow to decodethe overlapping neural signals evoked by stimuli presented close in time (e.g., Grootswagers et al., 2019a; Marti andDehaene, 2017; Robinson et al., 2019), even when these stimuli are not task-relevant (Grootswagers et al., 2019b;Marti and Dehaene, 2017; Robinson et al., 2019). Multivariate decoding methods can therefore be used to disentangleinformation from simultaneously presented stimuli and investigate the temporal dynamics of attentional mechanismsoperating on the stimuli.We conducted two experiments to investigate the e ect of attention on the representations of simultaneouslypresented objects and letters. Participants were shown images of objects overlaid with alphabet letters, or vice versa,in rapid succession and performed a cognitively demanding 2-back task on either the object or the letters, whichrequired attending to one of the two simultaneously presented stimuli. We then performed a multivariate decodinganalysis on all non-target object and letter stimuli in the presentation streams and examined the di erences betweenthe two task conditions. In both experiments, we found that we could decode all stimuli regardless of whether they rootswagers et al. 3 were attended, but that attention enhanced the decodability of the relevant stimulus (object versus letter). In Experi-ment 1, with small letters overlaid on larger objects, attentional e ects emerged around 220ms post-stimulus onsetfor objects, but for letters the di erence started earlier, at 100ms post-stimulus onset. In a second experiment, weexchanged the position of the stimuli on the display (i.e., letters overlaid with objects) and found that the timing dif-ference reversed accordingly. Our results show how early neural responses to simultaneously presented stimuli aremodulated by certain aspects of the stimulus (e.g., size of attended stimulus) as well as our current task and attentionalfocus. | METHODS
We performed two experiments that investigated the e ect of attention on the representations of non-target stimuliduring rapid serial visual presentation streams. Unless stated otherwise, the description of the methods below appliesto both experiments. Stimuli, code, and data for this study can be found at https://osf.io/ zhwp/ . | Stimuli and design
Stimuli consisted of 16 visual objects and 16 uppercase letters (ABCDEFGJKLQRTUVY). The visual objects werecoloured segmented objects obtained from spanning four categories (birds, sh, boats, and planes)with 4 images in each category. The categories could also be assigned to a two-way superordinate organisation(i.e., animals versus vehicles). The experiment was programmed in psychopy (Peirce et al., 2019). In Experiment 1,we superimposed one of 16 uppercase letters (approx. 0.8 degrees visual angle) in white font on a black circularbackground (Figure 1B&C) on top of the visual object stimuli (approx. 3.3 degrees visual angle). In Experiment 2,we superimposed the visual object stimuli (approx. 1.7 degrees visual angle) on one of the 16 uppercase letters(approx. 3.3 degrees visual angle) in white font on a black circular background (Figure 1D&E). Stimuli were presentedin sequences of 36 (two repeats of each stimulus plus two two-back targets) for 200ms each, followed by a blankscreen for 200ms. In other words, using a 2.5Hz presentation rate and a 50% duty-cycle. In alternating sequencesof stimuli, participants were instructed to attend the objects or the letters and perform a cognitively demanding two-back task. Participants pressed a button whenever the stimulus they were attending to (object or letter) was the sameas the stimulus that appeared two images beforehand.We constructed 48 sequences of 32 simultaneous object and letter combinations. A sequence of stimuli was con-structed by concatenating two sets of random permutations of 16 items (representing the stimuli), with the constraintthat there were no repeats amongst the middle 8 items. We selected two random positions for target placement, onein the rst half of each sequence and one in the second half of each sequence and inserted a target before and afterthe chosen positions, thus creating two-back repeats. The targets were never the same as the nearest three stimuli.Each stimulus was a target equally often. The order of stimuli in each sequence was mirror-copied, so that the order ofobjects and letters had matching properties while having targets in di erent positions. The 48 sequences were thenpresented twice in the experiment in random order (96 sequences in total), once for the object task, and once forthe letter task. The task condition of the rst sequence was counterbalanced across participants, and the conditionsalternated every sequence. Grootswagers et al.
F I G U R E 1
Stimuli and design. A) Stimuli were 16 segmented objects spanning four categories (birds, sh, boats,planes) and two superordinate categories (animals and vehicles). Stimuli were presented in sequences at 2.5Hz(200ms on, 200ms o ) and in each sequence, participants performed a two-back task on either the objects or on theletters. B) In the object task, participants responded with a button press when an object image was the same as thesecond-to-last image (two-back), while ignoring the letters. C) In the letter task, participants ignored the objectimages and responded on a two-back letter repeat. D, E) In the second experiment, the position of the letter andobjects were swapped while keeping all other details the same. rootswagers et al. 5 | EEG recordings and preprocessing
Participants in Experiment 1 were 20 adults (9 female, 11 male; mean age 24.45 years; age range 19-41 years; allright-handed). Participants in Experiment 2 were 20 adults (17 female, 3 male; mean age 22.45 years; age range19-36 years; 1 left-handed). All participants reported normal or corrected-to-normal vision and were recruited fromthe University of Sydney in return for payment or course credit. The study was approved by the University of Sydneyethics committee and informed consent was obtained from all participants. During EEG setup, participants practicedon example sequences of the two-back task. We used conductive gel to reduce impedance at each electrode sidebelow 10 kOhm where possible. The median electrode impedance was under 10 kOhm in 35/40 subjects and under 40kOhm in all subjects. Continuous EEG data were recorded from 64 electrodes arranged according to the internationalstandard 10–10 system for electrode placement (Jasper, 1958; Oostenveld and Praamstra, 2001) using a BrainVisionActiChamp system, digitized at a 1000-Hz sample rate (resolution: 0.0488281 µ V). Scalp electrodes were referencedonline to Cz. Event triggers were sent from the stimulus computer to the EEG ampli er using the parallel port. Weused the same preprocessing pipeline as our earlier work that applied MVPA to rapid serial visual processing paradigms(Grootswagers et al., 2019a,b; Robinson et al., 2019). Preprocessing was performed o ine using EEGlab (Delormeand Makeig, 2004). Data were ltered using a Hamming windowed FIR lter with 0.1Hz highpass and 100Hz lowpass lters, re-referenced to an average reference, and were downsampled to 250Hz. As in our previous work, no furtherpreprocessing steps were applied (e.g., baseline correction or epoch rejection), and the channel voltages at each timepoint were used for the remainder of the analysis. Epochs were created for each stimulus presentation ranging from[-100 to 1000ms] relative to stimulus onset. Target epochs (task-relevant two-back events) were excluded from theanalysis. | Decoding analysis
To assess the representations of attended and unattended stimuli in the neural signal, we applied an MVPA decod-ing pipeline (Grootswagers et al., 2017) to the EEG channel voltages. The decoding analyses were implemented inCoSMoMVPA (Oosterhof et al., 2016). A regularised ( = 0.01) linear discriminant analysis classi er was used in com-bination with an exemplar-by-sequence-cross-validation approach. Decoding was performed within subject, and thesubject-averaged results were analysed at the group level. This pipeline was applied to each stimulus in the sequenceto investigate object representations in fast sequences under di erent task requirements. For all sequences, we de-coded the 16 di erent object images, and the 16 di erent letters. We averaged over all pairwise decoding accuracies(i.e., bird 1 vs sh 1, bird 1 vs boat 4, bird 1 vs plane 1 etc.), such that chance-level was 50%. The analysis wasperformed separately for sequences from the two conditions (object task and letter task), resulting in a total of fourtime-varying decoding series of data per participant. For these analyses, we used a leave-one-sequence-out cross-validation scheme, where all epochs from one sequence were used as test set. We report the mean cross-validateddecoding accuracies.To determine the e ect of attention on higher-level image processing, we also decoded the category (bird, sh,boat, plane) and animacy (animal versus vehicle) of the visual objects. For these categorical contrasts, we used animage-by-sequence-cross-validation scheme so that identical images were not part of both training and test set (Carl-son et al., 2013; Grootswagers et al., 2019a, 2017). This was implemented by holding out one image from each cate-gory in one sequence as test data and training the classi er on the remaining images from the remaining sequences.This was repeated for all possible held-out pairs and held out sequences. The analyses were performed separately forthe object and letter conditions. Grootswagers et al. | Exploratory channel-searchlight
We performed an exploratory channel-searchlight analysis to further investigate which features (channels) of the EEGsignal were driving the classi cation accuracies. For each EEG channel, a local cluster was constructed by taking theclosest four neighbouring channels, and the decoding analyses were performed on the signal of only these channels.The decoding accuracies were stored at the centre channel of the cluster. This resulted in a time-by-channel map ofdecoding for each of the contrasts, and for each subject. | Statistical inference
We assessed whether stimulus information was present in the EEG signal by comparing classi cation accuracies tochance-level. To determine evidence for above chance decoding and evidence for di erences in decoding accuraciesbetween conditions we computed Bayes factors (Dienes, 2011; Je reys, 1961; Rouder et al., 2009; Wagenmakers,2007). For the alternative hypothesis of above-chance decoding, a JZS prior was used with default scale factor 0.707(Je reys, 1961; Rouder et al., 2009; Wetzels and Wagenmakers, 2012; Zellner and Siow, 1980). The prior for the nullhypothesis was set at chance level. We then calculated the Bayes factor (BF), which is the probability of the data underthe alternative hypothesis relative to the null hypothesis. For visualisation, we thresholded BF > 10 as substantialevidence for the alternative hypothesis, and BF < 1/3 as substantial evidence in favour of the null hypothesis (Je reys,1961; Wetzels et al., 2011). In addition, we computed frequentist statistics for decoding against chance, and fortesting for non-zero di erences in decoding accuracies. At each time point, a Wilcoxon sign-rank test was performedfor decoding accuracies against chance (one-tailed), and for the di erence between conditions (two-tailed). To correctfor multiple comparisons across time points, we computed FDR-adjusted p-values (Benjamini and Hochberg, 1995;Yekutieli and Benjamini, 1999). F I G U R E 2
Behavioural performance was similar between the object and letter tasks. A) Hit rate for all subjectsin Experiment 1 de ned as the proportion of correctly identi ed 2-back events. B) Hit rate for all subjects inExperiment 2. Bars show mean and standard error. Each dot represents the hit rate of one subject in one condition(object or letter task). Overall, Bayes Factors (displayed above the x-axis) indicated evidence for better performanceon the letter tasks. rootswagers et al. 7 F I G U R E 3 Di erent e ects of attention on decoding performance for objects and letters. Plots show decodingperformance over time for object decoding (A&B) and letter decoding (C&D). Di erent lines in each plot showdecoding accuracy during di erent tasks over time relative to stimulus onset, with shaded areas showing standarderror across subjects (N = 20). Their time-varying topographies are shown below each plot, averaged across 100mstime bins. Thresholded Bayes factors (BF) and p-values for above-chance decoding or non-zero di erences aredisplayed under each plot (note: di erence lines and topographies are shown together in Figure 4). For both objectsand letters, decodability was higher when they were task-relevant, but the respective time-courses of thesedi erences varied. Grootswagers et al.
F I G U R E 4
Aggregating the attention e ect over the two experiments shows the interaction between task and(relative) stimulus size. Plots shows the di erence in decoding performance between task-relevant andtask-irrelevant object decoding (A) and letter decoding (B). Each line re ects the mean di erence from one of thetwo experiments relative to stimulus onset, with shaded areas showing standard error across subjects (N = 20). Theirtime-varying topographies are shown below each plot, averaged across 50ms time bins. Thresholded Bayes factors(BF) and p-values for above-chance decoding or non-zero di erences are displayed under each plot. Note that theseare the same as the stats for the non-zero di erence in Figure 3. For both objects and letter stimuli, the onsets ofthe task-e ect (relevant-irrelevant) were earlier when the stimulus was smaller. rootswagers et al. 9 | RESULTS
We examined the temporal dynamics of visual processing for attended (task relevant) versus unattended (task ir-relevant) stimuli that were spatially and temporally overlapping. Participants performed a di cult two-back targetdetection task on objects or letters simultaneously presented at xation, and had to respond with a button press totwo-back events within 5 subsequent stimulus presentations (i.e., 2 seconds) for the response to be counted as correct.Behavioural results showed that participants performed reasonably well and, on average, correctly detected 52.29%(SE 3.52) of the two-back events in Experiment 1 (Figure 2A), and 61.59% (SE 3.97) of the two-back events in Exper-iment 2 (Figure 2B). Behavioural performance was similar for detection of object (mean 50.73%, SE 2.82) and letter(mean 53.85%, SE 4.24) targets in Experiment 1 (Figure 2A) and higher for the letter (mean 65.89%, SE 4.28) thanthe object (mean 57.29%, SE 3.67) targets in Experiment 2 (Figure 2B). Bayes Factors indicated weak evidence for nodi erence in performance between task contexts in Experiment 1 (Figure 2A), and evidence for better performanceon the letter task in Experiment 2 (Figure 2B).To investigate the temporal dynamics of processing for attended and unattended stimuli, we decoded the objectimages and letters in every sequence, separately for the object task and letter task sequences. Figure 3 shows that ob-jects and letters were decodable regardless of whether the stimulus was attended or not, but that attention enhancedboth object and letter processing. For objects, decodability was higher in the object task (task-relevant) relative tothe letter task (task-irrelevant), an e ect that emerged after 200ms and remained until around 600ms (Figure 3A).For letter decoding, performance was higher for the letter task than for the object task from 100ms to approximately600ms (Figure 3C).In Experiment 2, we exchanged the position of the object and letters on the screen, so that the letters werepresented larger and overlaid with a small object at xation. Here, attention similarly a ected object and letter pro-cessing, but attention e ects occurred at di erent times. In experiment 2, the attention e ect for objects emergingafter 180ms and remained until approximately 600ms (Figure 3B), and for letters occurred from 220ms to around600ms (Figure 3D). To integrate the results from both experiments, Figure 4 shows the e ect of attention for objectsand letters in both experiments (i.e., the di erences between decoding accuracies from Figure 3).Combining the results from both experiments (summarised in Figure 5) shows that the attention e ect startedearlier for the smaller item in the display. That is, the attention e ect on small letters started 100ms earlier than largeletters, and the attention e ect for small objects started 50ms earlier than large objects (Figure 5B). This suggeststhat mechanisms for attentional prioritisation are modulated by the relative retinal projection of the stimulus. Theexploratory channel searchlight for object decoding (Figure 4A) suggested that the stronger decoding in the attendedcondition was right-lateralised. Letter decoding channel searchlights (Figure 4B) showed a more left-lateralised dif-ference in the attended condition. Together, the channel-searchlight analyses suggest that attentional e ects werelateralised di erently between objects and letters, but these results should be interpreted with caution as channellocations can contain information from distant sites.To assess the e ect of attention on higher-level processes, we also performed object category decoding (e.g., birdversus sh) and animacy decoding (animals versus vehicles). For both contrasts, decodable information was evident inthe neural signal when objects were both attended and unattended, but information was more decodable when theywere attended. Figure 6 shows that animacy and category decoding were higher for the object task compared withthe letter task. Animacy (animal versus vehicle) was more decodable during the object task than the letter task fromapproximately 450-550ms in Experiment 1 (Figure 6A) and around 300ms in Experiment 2 (Figure 6B). In both Exper-iments, object category (e.g., bird versus sh) was more decodable from approximately 200ms (Figure 6C-D). Theseresults show increased decoding accuracy for the more abstract categorical object information when the stimulus was relevant for the current task, but with di erential e ects of attention depending on the size of the stimuli.These results are striking in showing that we can observe speci c patterns of activity for the letter and objectstimuli. Using a simple design, we show a clear e ect of attentional enhancement on object representations that variesby the size of the images. We note that it is unlikely that the decoding results are due to artefacts (e.g., noise, eye blinks,or saccades) in the EEG signal. Firstly, for artefacts to contribute to the decoding, they would have to systematicallycovary with the randomly ordered 16 di erent stimuli over the whole experiment in every subject, which is unlikely.Secondly, stimuli were presented at very short durations at xation, eliminating any bene t saccades would have ontask performance. Finally, if the information were linked to eye movements, we would expect to see information infrontal sensors, but the channel-searchlight showed information most prominently originated in occipital sensors. F I G U R E 5
Summary of main ndings. The top row (A,B) shows the signi cant time points for each contrast. Thebottom row (C,D) shows the time of the peak (denoted by x) accompanied by the distribution of peak times obtainedby drawing 10,000 samplings from the subjects with replacement. Left columns (A,C) show results for decodingagainst chance, and right columns (B,D) show the di erence between attended and unattended decoding. rootswagers et al. 11 F I G U R E 6 E ect of attention on higher level categorical object contrasts in Experiment 1 were similar toindividual object decoding. Plots show decoding performance over time for object animacy decoding (A) and objectcategory decoding (B). Di erent lines in each plot show decoding accuracy during di erent tasks over time relativeto stimulus onset, with shaded areas showing standard error across subjects (N = 20). Their time-varyingtopographies are shown below each plot, averaged across 100ms time bins. Thresholded Bayes factors (BF) andp-values for above-chance decoding or non-zero di erences are displayed under each plot. | DISCUSSION
In this study, we asked how attention modulates the representations of visual stimuli. Participants monitored streamsof letters overlaid on objects (Experiment 1) or objects overlaid on letters (Experiment 2) and performed a 2-backtarget detection task on either the letters or the objects. Importantly, we did not analyse the responses to the 2-backtargets, but rather investigated how task context in uenced the representation of all other stimuli in the streams. Wepredicted that attended and relevant information would be strongly a ected by attentional prioritisation processes inthe di cult 2-back task. Remarkably, we could decode all attended and unattended stimuli in both experiments, eventhough they were spatially and temporally overlapping, but the dynamics of the representations varied according tothe task and the size of the stimuli. As predicted, we found in general that attending to objects improved the decodingaccuracy for objects and that attending to letters improved the letter decoding accuracy. However, the time course ofthese attentional e ects varied, such that the improved decoding of task-relevant information emerged after 200msfor large stimuli, but before 200ms for small stimuli (Figure 5). Taken together, these ndings show that task contextselectively enhances the processing of relevant visual stimuli, and that this e ect is speci c to the features of thestimuli being selected.Our results shed light on the time course of attention and on the ongoing debate in the literature about whichstages of processing are a ected by attentional mechanisms (cf. Alilovi et al., 2019; Baumgartner et al., 2018). Theobservation that size a ects the temporal dynamics of attention on early representations is consistent with previousstudies showing early e ects when varying the spatial aspects of the task (Mangun, 1995; Wyart et al., 2015). Incontrast, previous work has used similar 2-back tasks on letters overlaid on images to divert attention away from theimages and found that the early processing of the images was not a ected (e.g., Groen et al., 2015). However, thiswork did not analyse the responses to the letters (i.e., the small items in their display), which our study showed to bea ected early in the response. The later and more prolonged e ects of attention on the decoding accuracies is con-sistent with work showing enhancements of target versus distractor (i.e., task-relevant stimulus information) codingstarting around 200ms (Kranczioch et al., 2005; Marti and Dehaene, 2017). These later (>200ms) sustained e ects areconsistent with work suggesting information processing up to 200ms is mainly driven by feedforward mechanisms,and that after around 200ms, a wider network of recurrent processes is recruited to generate behavioural decisions(Dehaene and Changeux, 2011; Gwilliams and King, 2020; Lamme and Roelfsema, 2000). Our results complementthese ndings by showing how attentional e ects interact with visual features of the stimulus (relative size), whichhighlights how attention can impact representations at di erent information processing stages. Future work can buildon our ndings by further investigating the interactions between early and late attentional e ects.All stimuli in this study evoked distinct patterns of neural responses regardless of whether they were relevant tothe task at hand. That is, letters and objects were decodable in all conditions. This ts with our previous work showingthat task-irrelevant objects can be decoded from rapid streams (Grootswagers et al., 2019a; Robinson et al., 2019),likely re ecting a degree of automaticity in visual processing and con rming that target selection is not a requirementfor stimulus processing. The current study extends these ndings by showing that two simultaneously presentedvisual objects are decodable even when one stimulus is much less prioritised than the other due to task demands andstimulus size. Strikingly, the duration of above chance decoding was much longer than the stimulus presentation time.The long, sustained decoding of attended stimuli could re ect the requirement of holding stimuli in memory for twosubsequent presentations (i.e., >800ms) to perform the 2-back task. An easier task with a smaller working memorycomponent may result in di erent attentional e ects, for example it may reduce the need for suppression of theirrelevant stimulus. However, the memory component of the task cannot fully account for the prolonged e ects, asfor objects, sustained above chance decoding was observed even when the stimulus was not attended, an observation rootswagers et al. 13 that is consistent with our previous work on information coding rapid sequences (Grootswagers et al., 2019a,b; Kingand Wyart, 2019; Robinson et al., 2019). For example, unattended object information was above chance for up to900ms post stimulus-onset in Experiment 1 (Figure 3A), and up to 600ms in Experiment 2, when the objects weresmaller (Figure 3B). This shows that visual information was maintained in the system even though it was not taskrelevant and it was presented in conjunction with a task-relevant stimulus. Thus, task-irrelevant information appearedto reach higher levels of processing than just feature processing, even though it was not the subject of attention.Indeed, category and animacy decoding (Figure 6) suggests that object stimuli were processed up to abstract levelsin the visual hierarchy. In sum, all objects and letters were decodable even during fast-changing visual input andeven when they were not attended. Importantly, however, we found that attention enhanced the distinctiveness (i.e.,decodability) of the attended visual stimuli.Attention a ected both the strength and duration of evoked visual representations. For both letters and objects,decodability was higher and prolonged when they were task-relevant compared to when they were irrelevant. This isparticularly striking because the letter and object tasks involved exactly the same sequences of images and analyses,so di erences in decoding arise exclusively from the attentional focus imposed by the task that participants performed.Furthermore, it is important to note that target images (i.e., the two repeating stimuli) were not analysed, meaningthat target selection and response processes were not contained within our results. The di erence we observedthus can mainly be attributed to attentional mechanisms. The enhancement of attended object information around220ms is consistent with evidence of e ects from the attentional blink and target selection literature, which has oftenreported di erences in N2 and P300 ERP components (Kranczioch et al., 2007, 2003; Sergent et al., 2005). Targetstimuli in rapid streams have been found to evoke stronger signals around 220ms (Marti and Dehaene, 2017). In thesedesigns, however, it is di cult to distinguish between the e ects of target-selection and the enhancement of task-relevant information. As all our analyses were performed on non-target stimuli, our results point towards a generalenhancement of task-relevant stimuli at this time scale, even for images that are not selected for target-processing.This points towards a more general enhancement e ect of task-relevant information occurring around 220ms thatsupports exible task performance in many paradigms.Attentional e ects on the letter stimuli followed a di erent trajectory to that of the objects, with an onset around100ms for letters versus 220ms for objects in Experiment 1. This could be explained by the letters comprising a smallerpart of the stimulus arrangement. Previous work has shown e ects of eccentricity on neural responses (e.g., Eimer,2000; Isik et al., 2014; Müller and Hübner, 2002), but our results could also be attributed to di erences in spatialattention allocated to the letter versus image task. Indeed, when we exchanged the stimulus position in Experiment2, we observed an earlier onset of the attentional e ects on object decoding, but the e ect for letters seemed tooccur later. Channel searchlight analyses further suggested that the attentional e ects were more left lateralisedfor the letter task, and right lateralised for the object task. The regions of highest decoding in the searchlights donot necessarily re ect the regions where the signal originates, but these results do t with previous work showingthat letter processing is typically left lateralised (Cohen et al., 2003; Puce et al., 1996), and that animate objects tendto evoke right hemisphere dominant responses (Bentin et al., 1996; Puce et al., 1996, 1995). The di erent spatio-temporal dynamics between the enhanced decodability of relevant information between the object and letter taskssuggest that attentional e ects are dependent on perceptual characteristics of the speci c stimuli being processed.For objects and their conceptual category decoding, we found evidence for no attentional e ect on the initial re-sponses (until around 180ms). This is consistent with recent work that reported no evidence for attentional e ects onearly visual ERP components or decoding accuracies (Alilovi et al., 2019; Baumgartner et al., 2018). In contrast, wedid nd attentional e ects on decoding accuracy for the earliest responses to letters (Figure 3C), which were moredecodable throughout the epochs when task relevant. One explanation of this di erence is that objects are auto- matically and unconsciously processed, but letters may require an active recruitment of their respective processingmechanisms. Alternatively, the object stimuli used here are visually much more distinct (di erent colours and shapes)than the letter stimuli which facilitates decoding of visual feature di erences.In conclusion, we found that attention enhances the representations of task-relevant visual stimuli, even whenthey were spatially and temporally overlapping with task-irrelevant stimuli, and even when the stimuli were not se-lected as target. Our results suggest that attentional e ects operate on the speci c perceptual processing mechanismsof the stimulus, di ering across stimulus type and size. This points towards a multi-stage implementation of informa-tion prioritisation that guides early perceptual processes, as well as later-stage mechanisms. Acknowledgements
This research was supported by ARC DP160101300 (TAC), ARC DP200101787 (TAC), and ARC DE200101159 (AKR).The authors acknowledge the University of Sydney HPC service for providing High Performance Computing resources.
Con ict of interest The authors declare no con icts of interests. References
Alilovi , J., Timmermans, B., Reteig, L. C., van Gaal, S. and Slagter, H. A. (2019) No evidence that predictions and attentionmodulate the rst feedforward sweep of cortical information processing. Cerebral Cortex .Baumgartner, H. M., Graulty, C. J., Hillyard, S. A. and Pitts, M. A. (2018) Does spatial attention modulate the earliest componentof the visual evoked potential?
Cognitive Neuroscience , , 4–19.Benjamini, Y. and Hochberg, Y. (1995) Controlling the false discovery rate: A practical and powerful approach to multipletesting. Journal of the Royal Statistical Society. Series B (Methodological) , , 289–300.Bentin, S., Allison, T., Puce, A., Perez, E. and McCarthy, G. (1996) Electrophysiological studies of face perception in humans. Journal of Cognitive Neuroscience , , 551–565.Carlson, T. A., Tovar, D. A., Alink, A. and Kriegeskorte, N. (2013) Representational dynamics of object vision: The rst 1000ms. Journal of Vision , , 1.Cohen, L., Martinaud, O., Lemer, C., Lehéricy, S., Samson, Y., Obadia, M., Slachevsky, A. and Dehaene, S. (2003) Visual wordrecognition in the left and right hemispheres: Anatomical and functional correlates of peripheral alexias. Cerebral Cortex , , 1313–1333.Dehaene, S. and Changeux, J.-P. (2011) Experimental and theoretical approaches to conscious processing. Neuron , , 200–227.Delorme, A. and Makeig, S. (2004) Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including indepen-dent component analysis. Journal of Neuroscience Methods , , 9–21.Dienes, Z. (2011) Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science , , 274–290.Ding, J., Sperling, G. and Srinivasan, R. (2006) Attentional modulation of ssvep power depends on the network tagged by the icker frequency. Cerebral Cortex , , 1016–1029. rootswagers et al. 15 Eimer, M. (2000) An erp study of sustained spatial attention to stimulus eccentricity.
Biological Psychology , , 205–220.Goddard, E., Carlson, T. A. and Woolgar, A. (2019) Spatial and feature-selective attention have distinct e ects on population-level tuning. bioRxiv , 530352.Groen, I. I. A., Ghebreab, S., Lamme, V. A. F. and Scholte, H. S. (2015) The time course of natural scene perception with reducedattention. Journal of Neurophysiology , , 931–946.Grootswagers, T., Robinson, A. K. and Carlson, T. A. (2019a) The representational dynamics of visual objects in rapid serialvisual processing streams. NeuroImage , , 668–679.Grootswagers, T., Robinson, A. K., Shatek, S. M. and Carlson, T. A. (2019b) Untangling featural and conceptual object repre-sentations. NeuroImage , , 116083.Grootswagers, T., Wardle, S. G. and Carlson, T. A. (2017) Decoding dynamic brain patterns from evoked responses: A tutorialon multivariate pattern analysis applied to time series neuroimaging data. Journal of Cognitive Neuroscience , , 677–697.Gwilliams, L. and King, J.-R. (2020) Recurrent processes support a cascade of hierarchical decisions. Elife , , e56603.Isik, L., Meyers, E. M., Leibo, J. Z. and Poggio, T. (2014) The dynamics of invariant object recognition in the human visualsystem. Journal of Neurophysiology , , 91–102.Jasper, H. H. (1958) The ten-twenty electrode system of the international federation. Electroencephalogr. Clin. Neurophysiol. , , 371–375.Je reys, H. (1961) Theory of probability . Oxford University Press.King, J.-R., Pescetelli, N. and Dehaene, S. (2016) Brain mechanisms underlying the brief maintenance of seen and unseensensory information.
Neuron , , 1122–1134.King, J.-R. and Wyart, V. (2019) The human brain encodes a chronicle of visual events at each instant of time. bioRxiv , 846576.Kranczioch, C., Debener, S. and Engel, A. K. (2003) Event-related potential correlates of the attentional blink phenomenon. Cognitive Brain Research , , 177–187.Kranczioch, C., Debener, S., Maye, A. and Engel, A. K. (2007) Temporal dynamics of access to consciousness in the attentionalblink. Neuroimage , , 947–955.Kranczioch, C., Debener, S., Schwarzbach, J., Goebel, R. and Engel, A. K. (2005) Neural correlates of conscious perception inthe attentional blink. Neuroimage , , 704–714.Lamme, V. A. F. and Roelfsema, P. R. (2000) The distinct modes of vision o ered by feedforward and recurrent processing. Trends in Neurosciences , , 571–579.Mangun, G. R. (1995) Neural mechanisms of visual selective attention. Psychophysiology , , 4–18.Mangun, G. R., Hillyard, S. A. and Luck, S. J. (1993) Electrocortical substrates of visual selective attention. In Attention andperformance 14: Synergies in experimental psychology, arti cial intelligence, and cognitive neuroscience , 219–243. Cambridge,MA, US: The MIT Press.Marti, S. and Dehaene, S. (2017) Discrete and continuous mechanisms of temporal selection in rapid visual streams. NatureCommunications , , 1955.Martinez-Trujillo, J. C. and Treue, S. (2004) Feature-based attention increases the selectivity of population responses in primatevisual cortex. Current Biology , , 744–751.Maunsell, J. H. and Treue, S. (2006) Feature-based attention in visual cortex. Trends in neurosciences , , 317–322. Mohsenzadeh, Y., Qin, S., Cichy, R. M. and Pantazis, D. (2018) Ultra-rapid serial visual presentation reveals dynamics of feed-forward and feedback processes in the ventral visual pathway. eLife , , e36329.Müller, M. M., Andersen, S., Trujillo, N. J., Valdés-Sosa, P., Malinowski, P. and Hillyard, S. A. (2006) Feature-selective attentionenhances color signals in early visual areas of the human brain. Proceedings of the National Academy of Sciences , ,14250–14254.Müller, M. M. and Hübner, R. (2002) Can the spotlight of attention be shaped like a doughnut? evidence from steady-statevisual evoked potentials. Psychological Science , , 119–124.Oostenveld, R. and Praamstra, P. (2001) The ve percent electrode system for high-resolution eeg and erp measurements. Clinical Neurophysiology , , 713–719.Oosterhof, N. N., Connolly, A. C. and Haxby, J. V. (2016) Cosmomvpa: Multi-modal multivariate pattern analysis of neuroimag-ing data in matlab/gnu octave. Frontiers in Neuroinformatics , .Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Kastman, E. and Lindeløv, J. K. (2019) Psychopy2:Experiments in behavior made easy. Behavior Research Methods , , 195–203.Puce, A., Allison, T., Asgari, M., Gore, J. C. and McCarthy, G. (1996) Di erential sensitivity of human visual cortex to faces,letterstrings, and textures: A functional magnetic resonance imaging study. Journal of Neuroscience , , 5205–5215.Puce, A., Allison, T., Gore, J. C. and McCarthy, G. (1995) Face-sensitive regions in human extrastriate cortex studied by func-tional mri. Journal of Neurophysiology , , 1192–1199.Robinson, A. K., Grootswagers, T. and Carlson, T. A. (2019) The in uence of image masking on object representations duringrapid serial visual presentation. NeuroImage , , 224–231.Rossion, B., Torfs, K., Jacques, C. and Liu-Shuang, J. (2015) Fast periodic presentation of natural images reveals a robustface-selective electrophysiological response in the human brain. Journal of Vision , , 18–18.Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D. and Iverson, G. (2009) Bayesian t tests for accepting and rejecting the nullhypothesis. Psychonomic bulletin & review , , 225–237.Rousselet, G. A., Fabre-Thorpe, M. and Thorpe, S. J. (2002) Parallel processing in high-level categorization of natural images. Nature Neuroscience , , 629–630.Sergent, C., Baillet, S. and Dehaene, S. (2005) Timing of the brain events underlying access to consciousness during theattentional blink. Nature Neuroscience , , 1391.Smout, C. A., Tang, M. F., Garrido, M. I. and Mattingley, J. B. (2019) Attention promotes the neural encoding of predictionerrors. PLOS Biology , , e2006812.Tang, M. F., Ford, L., Arabzadeh, E., Enns, J. T., Visser, T. A. W. and Mattingley, J. B. (2020) Neural dynamics of the attentionalblink revealed by encoding orientation selectivity during rapid visual presentation. Nature Communications , , 434.Wagenmakers, E.-J. (2007) A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review , ,779–804.Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J. and Wagenmakers, E.-J. (2011) Statistical evidence in experi-mental psychology: An empirical comparison using 855 t tests. Perspectives on Psychological Science , , 291–298.Wetzels, R. and Wagenmakers, E.-J. (2012) A default bayesian hypothesis test for correlations and partial correlations. Psy-chonomic Bulletin & Review , , 1057–1064.Wyart, V., Myers, N. E. and Summer eld, C. (2015) Neural mechanisms of human perceptual choice under focused and dividedattention. Journal of Neuroscience , , 3485–3498. rootswagers et al. 17 Yekutieli, D. and Benjamini, Y. (1999) Resampling-based false discovery rate controlling multiple test procedures for correlatedtest statistics.
Journal of Statistical Planning and Inference , , 171–196.Zellner, A. and Siow, A. (1980) Posterior odds ratios for selected regression hypotheses. 585–603. Valencia: University ofValencia Press.Zhang, W. and Luck, S. J. (2009) Feature-based attention modulates feedforward visual processing. Nature Neuroscience ,12