Najib J. Majaj

The role of spatial frequency channels in letter identification

How we see is today explained by physical optics and retinal transduction, followed by feature detection, in the cortex, by a bank of parallel independent spatial-frequency-selective channels. It is assumed that the observer uses whichever channels are best for the task at hand. Our current results demand a revision of this framework: Observers are not free to choose which channels they use. We used critical-band masking to characterize the channels mediating identification of broadband signals: letters in a wide range of fonts (Sloan, Bookman, Künstler, Yung), alphabets (Roman and Chinese), and sizes (0.1-55 degrees ). We also tested sinewave and squarewave gratings. Masking always revealed a single channel, 1.6+/-0.7 octaves wide, with a center frequency that depends on letter size and alphabet. We define an alphabets stroke frequency as the average number of lines crossed by a slice through a letter, divided by the letter width. For sharp-edged (i.e. broadband) signals, we find that stroke frequency completely determines channel frequency, independent of alphabet, font, and size. Moreover, even though observers have multiple channels, they always use the same channel for the same signals, even after hundreds of trials, regardless of whether the noise is low-pass, high-pass, or all-pass. This shows that observers identify letters through a single channel that is selected bottom-up, by the signal, not top-down by the observer. We thought shape would be processed similarly at all sizes. Bandlimited signals conform more to this expectation than do broadband signals. Here, we characterize processing by channel frequency. For sinewave gratings, as expected, channel frequency equals sinewave frequency f(channel)=f. For bandpass-filtered letters, channel frequency is proportional to center frequency f(channel) proportional, variantf(center) (log-log slope 1) when size is varied and the band (c/letter) is fixed, but channel frequency is less than proportional to center frequency f(channel) proportional, variantf(center)(2/3) (log-log slope 2/3) when the band is varied and size is fixed. Finally, our main result, for sharp-edged (i.e. broadband) letters and squarewaves, channel frequency depends solely on stroke frequency, f(channel)/10c/deg=(2/3), with a log-log slope of 2/3. Thus, large letters (and coarse squarewaves) are identified by their edges; small letters (and fine squarewaves) are identified by their gross strokes.

Are faces processed like words? A diagnostic test for recognition by parts

Do we identify an object as a whole or by its parts? This simple question has been surprisingly hard to answer. It has been suggested that faces are recognized as wholes and words are recognized by parts. Here we answer the question by applying a test for crowding. In crowding, a target is harder to identify in the presence of nearby flankers. Previous work has described crowding between objects. We show that crowding also occurs between the parts of an object. Such internal crowding severely impairs perception, identification, and fMRI face-area activation. We apply a diagnostic test for crowding to a word and a face, and we find that the critical spacing of the parts required for recognition is proportional to distance from fixation and independent of size and kind. The critical spacing defines an isolation field around the target. Some objects can be recognized only when each part is isolated from the rest of the object by the critical spacing. In that case, recognition is by parts. Recognition is holistic if the observer can recognize the object even when the whole object fits within a critical spacing. Such an object has only one part. Multiple parts within an isolation field will crowd each other and spoil recognition. To assess the robustness of the crowding test, we manipulated familiarity through inversion and the face- and word-superiority effects. We find that threshold contrast for word and face identification is the product of two factors: familiarity and crowding. Familiarity increases sensitivity by a factor of x1.5, independent of eccentricity, while crowding attenuates sensitivity more and more as eccentricity increases. Our findings show that observers process words and faces in much the same way: The effects of familiarity and crowding do not distinguish between them. Words and faces are both recognized by parts, and their parts -- letters and facial features -- are recognized holistically. We propose that internal crowding be taken as the signature of recognition by parts.

Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition

The primate visual system achieves remarkable visual object recognition performance even in brief presentations, and under changes to object exemplar, geometric transformations, and background variation (a.k.a. core visual object recognition). This remarkable performance is mediated by the representation formed in inferior temporal (IT) cortex. In parallel, recent advances in machine learning have led to ever higher performing models of object recognition using artificial deep neural networks (DNNs). It remains unclear, however, whether the representational performance of DNNs rivals that of the brain. To accurately produce such a comparison, a major difficulty has been a unifying metric that accounts for experimental limitations, such as the amount of noise, the number of neural recording sites, and the number of trials, and computational limitations, such as the complexity of the decoding classifier and the number of classifier training examples. In this work, we perform a direct comparison that corrects for these experimental limitations and computational considerations. As part of our methodology, we propose an extension of “kernel analysis” that measures the generalization accuracy as a function of representational complexity. Our evaluations show that, unlike previous bio-inspired models, the latest DNNs rival the representational performance of IT cortex on this visual object recognition task. Furthermore, we show that models that perform well on measures of representational performance also perform well on measures of representational similarity to IT, and on measures of predicting individual IT multi-unit responses. Whether these DNNs rely on computational mechanisms similar to the primate visual system is yet to be determined, but, unlike all previous bio-inspired models, that possibility cannot be ruled out merely on representational performance grounds.

Dynamics of motion signaling by neurons in macaque area MT

Most neurons in macaque area MT are selective for the direction of stimulus motion. By comparing direction selectivity for gratings and plaids, we classified MT neurons as pattern direction selective (PDS) or component direction selective (CDS). We compared the time course of responses in CDS and PDS neurons in opiate-anesthetized macaques, using a rapid pseudorandom sequence of gratings and plaids that moved in different directions. On average, responses began 6 ms earlier in CDS neurons than in PDS neurons. More importantly, the pattern-selective responses of PDS neurons did not reach their fully selective state until 50–75 ms after the responses of CDS neurons had stabilized. The population motion response of MT is therefore initially dominated by component motion signals, and does not completely represent pattern motion until substantially later. The circuits that compute pattern motion take more time to finish their work than those signaling component motion.

Motion Integration by Neurons in Macaque MT Is Local, Not Global

Direction-selective neurons in primary visual cortex have small receptive fields that encode the motions of local features. These motions often differ from the motion of the object to which they belong and must therefore be integrated elsewhere. A candidate site for this integration is visual cortical area MT (V5), in which cells with large receptive fields compute the motion of patterns. Previous studies of motion integration in MT have used stimuli that fill the receptive field, and thus do not test whether motion information is really integrated across this whole area. For each MT neuron, we identified two regions (“patches”) within the receptive field that were approximately equally effective in driving responses. We then measured responses to plaids whose component gratings overlapped within a patch, and compared them with responses to the same component gratings presented in separate patches. Cells that were selective for the direction of motion of the whole pattern when the gratings overlapped lost this selectivity when the gratings were separated and became selective instead for the direction of motion of the individual components. If MT cells simply pooled all of the inputs that endow them with a receptive field, they would encode all of the motions in the receptive field as belonging to a single object. Our results indicate instead that critical elements of the computations underlying pattern-direction selectivity in MT are done locally, on a scale smaller than the whole receptive field.

Grouping in object recognition: The role of a Gestalt law in letter identification

The Gestalt psychologists reported a set of laws describing how vision groups elements to recognize objects. The Gestalt laws “prescribe for us what we are to recognize ‘as one thing’” (Köhler, 1920). Were they right? Does object recognition involve grouping? Tests of the laws of grouping have been favourable, but mostly assessed only detection, not identification, of the compound object. The grouping of elements seen in the detection experiments with lattices and “snakes in the grass” is compelling, but falls far short of the vivid everyday experience of recognizing a familiar, meaningful, named thing, which mediates the ordinary identification of an object. Thus, after nearly a century, there is hardly any evidence that grouping plays a role in ordinary object recognition. To assess grouping in object recognition, we made letters out of grating patches and measured threshold contrast for identifying these letters in visual noise as a function of perturbation of grating orientation, phase, and offset. We define a new measure, “wiggle”, to characterize the degree to which these various perturbations violate the Gestalt law of good continuation. We find that efficiency for letter identification is inversely proportional to wiggle and is wholly determined by wiggle, independent of how the wiggle was produced. Thus the effects of three different kinds of shape perturbation on letter identifiability are predicted by a single measure of goodness of continuation. This shows that letter identification obeys the Gestalt law of good continuation and may be the first confirmation of the original Gestalt claim that object recognition involves grouping.

Functional Segregation of Cortical Regions Underlying Speech Timing and Articulation

Spoken language is a central part of our everyday lives, but the precise roles that individual cortical regions play in the production of speech are often poorly understood. To address this issue, we focally lowered the temperature of distinct cortical regions in awake neurosurgical patients, and we relate this perturbation to changes in produced speech sequences. Using this method, we confirm that speech is highly lateralized, with the vast majority of behavioral effects seen on the left hemisphere. We then use this approach to demonstrate a clear functional dissociation between nearby cortical speech sites. Focal cooling of pars triangularis/pars opercularis (Brocas region) and the ventral portion of the precentral gyrus (speech motor cortex) resulted in the manipulation of speech timing and articulation, respectively. Our results support a class of models that have proposed distinct processing centers underlying motor sequencing and execution for speech.

Binocular integration of pattern motion signals by MT neurons and by human observers

Analysis of the movement of a complex visual stimulus is expressed in the responses of pattern-direction-selective neurons in area MT, which depend in turn on directionally selective inputs from area V1. How do MT neurons integrate their inputs? Pattern selectivity in MT breaks down when the gratings comprising a moving plaid are presented to non-overlapping regions of the (monocular) receptive field. Here we ask an analogous question, is pattern selectivity maintained when the component gratings are presented dichoptically to binocular MT neurons? We recorded from single units in area MT, measuring responses to monocular gratings and plaids, and to dichoptic plaids in which the components are presented separately to each eye. Neurons that are pattern selective when tested monocularly lose this selectivity when stimulated with dichoptic plaids. When human observers view these same stimuli, dichoptic plaids induce binocular rivalry. Yet motion signals from each eye can be integrated despite rivalry, revealing a dissociation of form and motion perception. These results reveal the role of monocular mechanisms in the computation of pattern motion in single neurons, and demonstrate that the perception of motion is not fully represented by the responses of individual MT neurons.

A New Code for Contrast in the Primate Visual Pathway

We characterize a hitherto undocumented type of neuron present in the regions bordering the principal layers of the macaque lateral geniculate nucleus. Neurons of this type were distinguished by a high and unusually regular maintained discharge that was suppressed by spatiotemporal modulation of luminance or chromaticity within the receptive field. The response to any effective stimulus was a reduction in discharge, reminiscent of the “suppressed-by-contrast” cells of the cat retina. To a counterphase-modulated grating, the response was a phase-insensitive suppression modulated at twice the stimulus frequency, implying a receptive field comprised of multiple mechanisms that generate rectifying responses. This distinctive nonlinearity makes the neurons well suited to computing a measure of contrast energy; such a signal might be important in regulating sensitivity early in visual cortex.

Population representation of visual information in areas V1 and V2 of amblyopic macaques

Amblyopia is a developmental disorder resulting in poor vision in one eye. The mechanism by which input to the affected eye is prevented from reaching the level of awareness remains poorly understood. We recorded simultaneously from large populations of neurons in the supragranular layers of areas V1 and V2 in 6 macaques that were made amblyopic by rearing with artificial strabismus or anisometropia, and 1 normally reared control. In agreement with previous reports, we found that cortical neuronal signals driven through the amblyopic eyes were reduced, and that cortical neurons were on average more strongly driven by the non-amblyopic than by the amblyopic eyes. We analyzed multiunit recordings using standard population decoding methods, and found that visual signals from the amblyopic eye, while weakened, were not degraded enough to explain the behavioral deficits. Thus additional losses must arise in downstream processing. We tested the idea that under monocular viewing conditions, only signals from neurons dominated by - rather than driven by - the open eye might be used. This reduces the proportion of neuronal signals available from the amblyopic eye, and amplifies the interocular difference observed at the level of single neurons. We conclude that amblyopia might arise in part from degradation in the neuronal signals from the amblyopic eye, and in part from a reduction in the number of signals processed by downstream areas.