Andreas Wutz

Temporal Integration Windows in Neural Processing and Perception Aligned to Saccadic Eye Movements

Summary When processing dynamic input, the brain balances the opposing needs of temporal integration and sensitivity to change. We hypothesized that the visual system might resolve this challenge by aligning integration windows to the onset of newly arriving sensory samples. In a series of experiments, human participants observed the same sequence of two displays separated by a brief blank delay when performing either an integration or segregation task. First, using magneto-encephalography (MEG), we found a shift in the stimulus-evoked time courses by a 150-ms time window between task signals. After stimulus onset, multivariate pattern analysis (MVPA) decoding of task in occipital-parietal sources remained above chance for almost 1 s, and the task-decoding pattern interacted with task outcome. In the pre-stimulus period, the oscillatory phase in the theta frequency band was informative about both task processing and behavioral outcome for each task separately, suggesting that the post-stimulus effects were caused by a theta-band phase shift. Second, when aligning stimulus presentation to the onset of eye fixations, there was a similar phase shift in behavioral performance according to task demands. In both MEG and behavioral measures, task processing was optimal first for segregation and then integration, with opposite phase in the theta frequency range (3–5 Hz). The best fit to neurophysiological and behavioral data was given by a dampened 3-Hz oscillation from stimulus or eye fixation onset. The alignment of temporal integration windows to input changes found here may serve to actively organize the temporal processing of continuous sensory input.

Temporal Windows in Visual Processing: “Prestimulus Brain State” and “Poststimulus Phase Reset” Segregate Visual Transients on Different Temporal Scales

Dynamic vision requires both stability of the current perceptual representation and sensitivity to the accumulation of sensory evidence over time. Here we study the electrophysiological signatures of this intricate balance between temporal segregation and integration in vision. Within a forward masking paradigm with short and long stimulus onset asynchronies (SOA), we manipulated the temporal overlap of the visual persistence of two successive transients. Human observers enumerated the items presented in the second target display as a measure of the informational capacity read-out from this partly temporally integrated visual percept. We observed higher β-power immediately before mask display onset in incorrect trials, in which enumeration failed due to stronger integration of mask and target visual information. This effect was timescale specific, distinguishing between segregation and integration of visual transients that were distant in time (long SOA). Conversely, for short SOA trials, mask onset evoked a stronger visual response when mask and targets were correctly segregated in time. Examination of the target-related response profile revealed the importance of an evoked α-phase reset for the segregation of those rapid visual transients. Investigating this precise mapping of the temporal relationships of visual signals onto electrophysiological responses highlights how the stream of visual information is carved up into discrete temporal windows that mediate between segregated and integrated percepts. Fragmenting the stream of visual information provides a means to stabilize perceptual events within one instant in time.

Dense sampling reveals behavioral oscillations in rapid visual categorization.

Perceptual systems must create discrete objects and events out of a continuous flow of sensory information. Previous studies have demonstrated oscillatory effects in the behavioral outcome of low-level visual tasks, suggesting a cyclic nature of visual processing as the solution. To investigate whether these effects extend to more complex tasks, a stream of “neutral” photographic images (not containing targets) was rapidly presented (20 ms/image). Embedded were one or two presentations of a randomly selected target image (vehicles and animals). Subjects reported the perceived target category. On dual-presentation trials, the ISI varied systematically from 0 to 600 ms. At randomized timing before first target presentation, the screen was flashed with the intent of creating a phase reset in the visual system. Sorting trials by temporal distance between flash and first target presentation revealed strong oscillations in behavioral performance, peaking at 5 Hz. On dual-target trials, longer ISIs led to reduced performance, implying a temporal integration window for object category discrimination. The “animal” trials exhibited a significant oscillatory component around 5 Hz. Our results indicate that oscillatory effects are not mere fringe effects relevant only with simple stimuli, but are resultant from the core mechanisms of visual processing and may well extend into real-life scenarios.

The temporal window of individuation limits visual capacity

One of the main tasks of vision is to individuate and recognize specific objects. Unlike the detection of basic features, object individuation is strictly limited in capacity. Previous studies of capacity, in terms of subitizing ranges or visual working memory, have emphasized spatial limits in the number of objects that can be apprehended simultaneously. Here, we present psychophysical and electrophysiological evidence that capacity limits depend instead on time. Contrary to what is commonly assumed, subitizing, the reading-out a small set of individual objects, is not an instantaneous process. Instead, individuation capacity increases in steps within the lifetime of visual persistence of the stimulus, suggesting that visual capacity limitations arise as a result of the narrow window of feedforward processing. We characterize this temporal window as coordinating individuation and integration of sensory information over a brief interval of around 100 ms. Neural signatures of integration windows are revealed in reset alpha oscillations shortly after stimulus onset within generators in parietal areas. Our findings suggest that short-lived alpha phase synchronization (≈1 cycle) is key for individuation and integration of visual transients on rapid time scales (<100 ms). Within this time frame intermediate-level vision provides an equilibrium between the competing needs to individuate invariant objects, integrate information about those objects over time, and remain sensitive to dynamic changes in sensory input. We discuss theoretical and practical implications of temporal windows in visual processing, how they create a fundamental capacity limit, and their role in constraining the real-time dynamics of visual processing.

Temporal buffering and visual capacity: The time course of object formation underlies capacity limits in visual cognition

Capacity limits are a hallmark of visual cognition. The upper boundary of our ability to individuate and remember objects is well known but—despite its central role in visual information processing—not well understood. Here, we investigated the role of temporal limits in the perceptual processes of forming “object files.” Specifically, we examined the two fundamental mechanisms of object file formation—individuation and identification—by selectively interfering with visual processing by using forward and backward masking with variable stimulus onset asynchronies. While target detection was almost unaffected by these two types of masking, they showed distinct effects on the two different stages of object formation. Forward “integration” masking selectively impaired object individuation, whereas backward “interruption” masking only affected identification and the consolidation of information into visual working memory. We therefore conclude that the inherent temporal dynamics of visual information processing are an essential component in creating the capacity limits in object individuation and visual working memory.

Frequency modulation of neural oscillations according to visual task demands

Significance Neural oscillations are hypothesized to play an important role in modulating perceptual processing in accordance with top-down goals. For instance, the amplitude, phase, and spatial distribution of alpha-band oscillations change with attention. Given recent links between the peak frequency of alpha oscillations and the temporal resolution of perception, we investigated whether frequency modulation occurs when task demands emphasize integration or segregation of visual input over time. We found that alpha frequency in occipital–temporal cortex decreased during, and in anticipation of, stimulus processing when task demands required temporal integration compared with segregation. These results demonstrate a unique top-down mechanism by which the brain controls the temporal resolution of visual processing in accordance with current task demands. Temporal integration in visual perception is thought to occur within cycles of occipital alpha-band (8–12 Hz) oscillations. Successive stimuli may be integrated when they fall within the same alpha cycle and segregated for different alpha cycles. Consequently, the speed of alpha oscillations correlates with the temporal resolution of perception, such that lower alpha frequencies provide longer time windows for perceptual integration and higher alpha frequencies correspond to faster sampling and segregation. Can the brain’s rhythmic activity be dynamically controlled to adjust its processing speed according to different visual task demands? We recorded magnetoencephalography (MEG) while participants switched between task instructions for temporal integration and segregation, holding stimuli and task difficulty constant. We found that the peak frequency of alpha oscillations decreased when visual task demands required temporal integration compared with segregation. Alpha frequency was strategically modulated immediately before and during stimulus processing, suggesting a preparatory top-down source of modulation. Its neural generators were located in occipital and inferotemporal cortex. The frequency modulation was specific to alpha oscillations and did not occur in the delta (1–3 Hz), theta (3–7 Hz), beta (15–30 Hz), or gamma (30–50 Hz) frequency range. These results show that alpha frequency is under top-down control to increase or decrease the temporal resolution of visual perception.

Rapid enumeration within a fraction of a single glance: The role of visible persistence in object individuation capacity

The number of items that can be individuated at a single glance is limited. Here, we investigate object individuation at a higher temporal resolution, in fractions of a single glance. In two experiments involving object individuation we manipulated the duration of visual persistence of the target items with a forward masking procedure. The number of items as well as their stimulus–onset asynchrony (SOA) to the mask was varied independently. The results showed main effects of numerosity and SOA, as well as an interaction. These effects were not caused by a generic reduction of item visibility by the mask. Instead, the SOA manipulation appeared to fractionate the time to access the sensory image. These findings suggest that the capacity limit of 3–4 items found in object individuation is, at least partially, the consequence of the temporal window of access to sensory information.

Expansion and Compression of Time Correlate with Information Processing in an Enumeration Task.

Perception of temporal duration is subjective and is influenced by factors such as attention and context. For example, unexpected or emotional events are often experienced as if time subjectively expands, suggesting that the amount of information processed in a unit of time can be increased. Time dilation effects have been measured with an oddball paradigm in which an infrequent stimulus is perceived to last longer than standard stimuli in the rest of the sequence. Likewise, time compression for the oddball occurs when the duration of the standard items is relatively brief. Here, we investigated whether the amount of information processing changes when time is perceived as distorted. On each trial, an oddball stimulus of varying numerosity (1–14 items) and duration was presented along with standard items that were either short (70 ms) or long (1050 ms). Observers were instructed to count the number of dots within the oddball stimulus and to judge its relative duration with respect to the standards on that trial. Consistent with previous results, oddballs were reliably perceived as temporally distorted: expanded for longer standard stimuli blocks and compressed for shorter standards. The occurrence of these distortions of time perception correlated with perceptual processing; i.e. enumeration accuracy increased when time was perceived as expanded and decreased with temporal compression. These results suggest that subjective time distortions are not epiphenomenal, but reflect real changes in sensory processing. Such short-term plasticity in information processing rate could be evolutionarily advantageous in optimizing perception and action during critical moments.

Object-based perception of orientation in the Ternus-Pikler display

Early visual processing is based on orientation-selective receptive fields in a retinotopic reference frame. Perception of visual features over longer time scales, exceeding this fast feedforward encoding, has been demonstrated to involve object-based, rather than only retinotopic coordinates (Fracasso et al., 2010; Boi et al., 2011). For example, non-retinotopic encoding has been found using the Ternus-Pikler (T-P) apparent motion display in which object identity is mapped across the object motion path given the inter-stimulus interval (ISI) is sufficiently long (Boi et al., 2009). Here, we report evidence that feature integration over time can involve an object-based frame of reference, even for the perhaps most paradigmatic example of retinotopically defined features: orientation. We presented observers with repeated series of T-P displays, in which the perceived rotation of Gabor patches depended on the combination of Gabor orientations in either retinotopic or object-based coordinates across display frames. We report that the frequency of perceived retinotopic rotations linearly decreases with increasing ISI between T-P display frames. For very short ISIs (< 50 ms) perceived rotation is strongly biased towards retinotopic processing but on longer time scales (exceeding ISIs around 100 ms) the rotation percept appears ambiguous or predominantly non-retinotopic for individual observers. In addition to these temporal factors, we show that in perceptually ambiguous T-P displays (constant ISI; 200 ms) the perceived rotation can be strongly biased towards either retinotopic or non-retinotopic integration based on object grouping. Cueing either static spatial object position or apparent motion resulted in robust element- or group motion percepts, respectively, and the frequency of retinotopic vs. non-retinotopic rotation reports depended strongly on the perceived object matching. Our results indicate that temporal integration of even basic, low-level visual features like orientation can be biased towards non-retinotopic processing in order to support the perceived constancy of objects in motion. Meeting abstract presented at VSS 2015.