Tomomi Fujimura

Categorical and dimensional perceptions in decoding emotional facial expressions

We investigated whether categorical perception and dimensional perception can co-occur while decoding emotional facial expressions. In Experiment 1, facial continua with endpoints consisting of four basic emotions (i.e., happiness–fear and anger–disgust) were created by a morphing technique. Participants rated each facial stimulus using a categorical strategy and a dimensional strategy. The results show that the happiness–fear continuum was divided into two clusters based on valence, even when using the dimensional strategy. Moreover, the faces were arrayed in order of the physical changes within each cluster. In Experiment 2, we found a category boundary within other continua (i.e., surprise–sadness and excitement–disgust) with regard to the arousal and valence dimensions. These findings indicate that categorical perception and dimensional perception co-occurred when emotional facial expressions were rated using a dimensional strategy, suggesting a hybrid theory of categorical and dimensional accounts.

Decision-Making Based on Emotional Images

The emotional outcome of a choice affects subsequent decision making. While the relationship between decision making and emotion has attracted attention, studies on emotion and decision making have been independently developed. In this study, we investigated how the emotional valence of pictures, which was stochastically contingent on participants’ choices, influenced subsequent decision making. In contrast to traditional value-based decision-making studies that used money or food as a reward, the “reward value” of the decision outcome, which guided the update of value for each choice, is unknown beforehand. To estimate the reward value of emotional pictures from participants’ choice data, we used reinforcement learning models that have successfully been used in previous studies for modeling value-based decision making. Consequently, we found that the estimated reward value was asymmetric between positive and negative pictures. The negative reward value of negative pictures (relative to neutral pictures) was larger in magnitude than the positive reward value of positive pictures. This asymmetry was not observed in valence for an individual picture, which was rated by the participants regarding the emotion experienced upon viewing it. These results suggest that there may be a difference between experienced emotion and the effect of the experienced emotion on subsequent behavior. Our experimental and computational paradigm provides a novel way for quantifying how and what aspects of emotional events affect human behavior. The present study is a first step toward relating a large amount of knowledge in emotion science and in taking computational approaches to value-based decision making.

Neural basis of decision-making guided by emotional outcomes

Emotional events resulting from a choice influence an individuals subsequent decision making. Although the relationship between emotion and decision making has been widely discussed, previous studies have mainly investigated decision outcomes that can easily be mapped to reward and punishment, including monetary gain/loss, gustatory stimuli, and pain. These studies regard emotion as a modulator of decision making that can be made rationally in the absence of emotions. In our daily lives, however, we often encounter various emotional events that affect decisions by themselves, and mapping the events to a reward or punishment is often not straightforward. In this study, we investigated the neural substrates of how such emotional decision outcomes affect subsequent decision making. By using functional magnetic resonance imaging (fMRI), we measured brain activities of humans during a stochastic decision-making task in which various emotional pictures were presented as decision outcomes. We found that pleasant pictures differentially activated the midbrain, fusiform gyrus, and parahippocampal gyrus, whereas unpleasant pictures differentially activated the ventral striatum, compared with neutral pictures. We assumed that the emotional decision outcomes affect the subsequent decision by updating the value of the options, a process modeled by reinforcement learning models, and that the brain regions representing the prediction error that drives the reinforcement learning are involved in guiding subsequent decisions. We found that some regions of the striatum and the insula were separately correlated with the prediction error for either pleasant pictures or unpleasant pictures, whereas the precuneus was correlated with prediction errors for both pleasant and unpleasant pictures.

Individual differences in heart rate variability are associated with the avoidance of negative emotional events.

Although the emotional outcome of a choice generally affects subsequent decisions, humans can inhibit the influence of emotion. Heart rate variability (HRV) has emerged as an objective measure of individual differences in the capacity for inhibitory control. In the present study, we investigated how individual differences in HRV at rest are associated with the emotional effects of the outcome of a choice on subsequent decision making using a decision-making task in which emotional pictures appeared as decision outcomes. We used a reinforcement learning model to characterize the observed behaviors according to several parameters, namely, the learning rate and the motivational value of positive and negative pictures. Consequently, we found that individuals with a lower resting HRV exhibited a greater negative motivational value in response to negative pictures, suggesting that these individuals tend to avoid negative pictures compared with individuals with a higher resting HRV.

Contextual Modulation of Physiological and Psychological Responses Triggered by Emotional Stimuli

A series of emotional events successively occur in temporal context. The present study investigated how physiological and psychological responses are modulated by emotional context. Skin conductance response (SCR), heart rate, corrugator activity, zygomatic activity, and subjective feelings during emotional picture viewing were measured. To create an emotional context, a unpleasant or pleasant picture was preceded by three types of pictures, i.e., unpleasant, pleasant, and neutral pictures, resulting in six pairings. The results showed that viewing an unpleasant picture attenuated pleasant feelings induced by the following pleasant picture. On the other hand, preceding pleasant pictures decreased SCR to the following pictures. The effects of contextual modulation on emotional responses might be due to the informative function of pre-existing feelings; unpleasant feelings signal a threatening environment, whereas pleasant feelings signal a benign environment. With respect to facial muscle activities, viewing a pleasant picture decreased corrugator activity in response to the preceding picture. These findings suggest several types of contextual modulation effects on psychological, autonomic, and somatic responses to emotional stimuli.

Event-Related Potentials Elicited by Pre-Attentive Emotional Changes in Temporal Context

The ability to detect emotional change in the environment is essential for adaptive behavior. The current study investigated whether event-related potentials (ERPs) can reflect emotional change in a visual sequence. To assess pre-attentive processing, we examined visual mismatch negativity (vMMN): the negative potentials elicited by a deviant (infrequent) stimulus embedded in a sequence of standard (frequent) stimuli. Participants in two experiments pre-attentively viewed visual sequences of Japanese kanji with different emotional connotations while ERPs were recorded. The visual sequence in Experiment 1 consisted of neutral standards and two types of emotional deviants with a strong and weak intensity. Although the results indicated that strongly emotional deviants elicited more occipital negativity than neutral standards, it was unclear whether these negativities were derived from emotional deviation in the sequence or from the emotional significance of the deviants themselves. In Experiment 2, the two identical emotional deviants were presented against different emotional standards. One type of deviants was emotionally incongruent with the standard and the other type of deviants was emotionally congruent with the standard. The results indicated that occipital negativities elicited by deviants resulted from perceptual changes in a visual sequence at a latency of 100–200 ms and from emotional changes at latencies of 200–260 ms. Contrary to the results of the ERP experiment, reaction times to deviants showed no effect of emotional context; negative stimuli were consistently detected more rapidly than were positive stimuli. Taken together, the results suggest that brain signals can reflect emotional change in a temporal context.

The implicit processing of categorical and dimensional strategies: an fMRI study of facial emotion perception

Our understanding of facial emotion perception has been dominated by two seemingly opposing theories: the categorical and dimensional theories. However, we have recently demonstrated that hybrid processing involving both categorical and dimensional perception can be induced in an implicit manner (Fujimura etal., 2012). The underlying neural mechanisms of this hybrid processing remain unknown. In this study, we tested the hypothesis that separate neural loci might intrinsically encode categorical and dimensional processing functions that serve as a basis for hybrid processing. We used functional magnetic resonance imaging to measure neural correlates while subjects passively viewed emotional faces and performed tasks that were unrelated to facial emotion processing. Activity in the right fusiform face area (FFA) increased in response to psychologically obvious emotions and decreased in response to ambiguous expressions, demonstrating the role of the FFA in categorical processing. The amygdala, insula and medial prefrontal cortex exhibited evidence of dimensional (linear) processing that correlated with physical changes in the emotional face stimuli. The occipital face area and superior temporal sulcus did not respond to these changes in the presented stimuli. Our results indicated that distinct neural loci process the physical and psychological aspects of facial emotion perception in a region-specific and implicit manner.

Modulation of Emotional Category Induced by Temporal Factors in Emotion Recognition

Categorical perception (CP), the perceptual experience whereby continuous sensory phenomena are perceived as distinct and separate percepts, is one of the most characteristic features of information processing in human cognition. CP is considered as the result of the integration of the top-down processing including background knowledge and verbal labeling and the bottom-up processing such as physical characteristics of the sensory signal. However, the underlying mechanisms governing the integration remain unclear. To address this issue, we focused on the temporal characteristics of CP of facial expression. In the current study, we investigated the contributions of temporal factors in CP processes, using facial expression recognition tasks as an example of CP. Participants completed an identification task and a discrimination task, well-established tasks for evaluating CP of facial expressions, with variable temporal parameters, that is, duration of stimulus presentation and delay time (interval between stimulus and response). The results demonstrated that the emotionally ambiguous stimuli are categorized more distinctively with the extension of delay length, not of stimulus duration. In contrast, the category boundary for facial expressions shifted toward “happy” with extention in stimulus duration, not in delay length. This dissociation between the impact of stimulus duration and delay suggests that there are two processes contributing to CP of facial emotion; one process may reflect the internal processing associated with the length of the delay period including verbal labeling of the stimuli, and the other process may reflect the temporal summation of stimulus inputs, associated with stimulus duration. These findings suggest that paying more attention to temporal factors in CP could be useful for further study of the mechanisms underlying CP.

Respect and admiration differentially activate the anterior temporal lobe

Admiration and respect are positive social emotions often experienced when recognizing excellent behavior in another person. Although both strongly rely on appraisal of behavior, admiration focuses on the admirable behavior of a person, while respect focuses on the person as a whole. The evaluation and interpretation of the social behavior of another person are dependent on semantic memory. Social semantic knowledge is represented in the anterior temporal lobe (ATL), and ATL activity is modulated by conceptual details of semantic knowledge. As respect requires evaluation of not only excellent behavior but also of the person as a whole, we hypothesized that the ATL is differentially activated by admiration and respect. To test our hypothesis, we conducted functional magnetic resonance imaging experiments. We presented participants with vignettes describing admirable behavior of fictitious characters and asked them to imagine and report how they would normally feel when encountering the situation described in the vignettes, i.e., admiration or respect and its intensity. A part of the left ATL was more strongly modulated by the intensity of respect than of admiration. Although admiration and respect are often considered to be closely related, our results indicate that the neural substrates underlying these emotions are different.

The functional role of facial expressions in communication

This paper overviews research on facial expressions focusing on its social functions in communication. The psychological origin of facial expressions has been discussed from two perspectives: expressions triggered by internal emotional states and expressions triggered by social contexts. These perspectives can also be applied to congruent facial expressions that automatically occur when observing others facial expressions. To date, research suggests that congruent facial expressions contribute to the understanding of others emotion and regulate affiliative relationships in social contexts, such as trustworthiness. The evidence for social functions of congruent facial expressions would be a key to understanding the physiological and evolutional basis of facial expressions in communication.