Jinhwan Kwon

The Effect of Visual Apparent Motion on Audiovisual Simultaneity

Visual motion information from dynamic environments is important in multisensory temporal perception. However, it is unclear how visual motion information influences the integration of multisensory temporal perceptions. We investigated whether visual apparent motion affects audiovisual temporal perception. Visual apparent motion is a phenomenon in which two flashes presented in sequence in different positions are perceived as continuous motion. Across three experiments, participants performed temporal order judgment (TOJ) tasks. Experiment 1 was a TOJ task conducted in order to assess audiovisual simultaneity during perception of apparent motion. The results showed that the point of subjective simultaneity (PSS) was shifted toward a sound-lead stimulus, and the just noticeable difference (JND) was reduced compared with a normal TOJ task with a single flash. This indicates that visual apparent motion affects audiovisual simultaneity and improves temporal discrimination in audiovisual processing. Experiment 2 was a TOJ task conducted in order to remove the influence of the amount of flash stimulation from Experiment 1. The PSS and JND during perception of apparent motion were almost identical to those in Experiment 1, but differed from those for successive perception when long temporal intervals were included between two flashes without motion. This showed that the result obtained under the apparent motion condition was unaffected by the amount of flash stimulation. Because apparent motion was produced by a constant interval between two flashes, the results may be accounted for by specific prediction. In Experiment 3, we eliminated the influence of prediction by randomizing the intervals between the two flashes. However, the PSS and JND did not differ from those in Experiment 1. It became clear that the results obtained for the perception of visual apparent motion were not attributable to prediction. Our findings suggest that visual apparent motion changes temporal simultaneity perception and improves temporal discrimination in audiovisual processing.

Detection of Nonverbal Synchronization through Phase Difference in Human Communication

Nonverbal communication is an important factor in human communication, and body movement synchronization in particular is an important part of nonverbal communication. Some researchers have analyzed body movement synchronization by focusing on changes in the amplitude of body movements. However, the definition of “body movement synchronization” is still unclear. From a theoretical viewpoint, phase difference is the most important factor in synchronization analysis. Therefore, there is a need to measure the synchronization of body movements using phase difference. The purpose of this study was to provide a quantitative definition of the phase difference distribution for detecting body movement synchronization in human communication. The phase difference distribution was characterized using four statistical measurements: density, mean phase difference, standard deviation (SD) and kurtosis. To confirm the effectiveness of our definition, we applied it to human communication in which the roles of speaker and listener were defined. Specifically, we examined the difference in the phase difference distribution between two different communication situations: face-to-face communication with visual interaction and remote communication with unidirectional visual perception. Participant pairs performed a task supposing lecture in the face-to-face communication condition and in the remote communication condition via television. Throughout the lecture task, we extracted a set of phase differences from the time-series data of the acceleration norm of head nodding motions between two participants. Statistical analyses of the phase difference distribution revealed the characteristics of head nodding synchronization. Although the mean phase differences in synchronized head nods did not differ significantly between the conditions, there were significant differences in the densities, the SDs and the kurtoses of the phase difference distributions of synchronized head nods. These results show the difference in nonverbal synchronization between different communication types. Our study indicates that the phase difference distribution is useful in detecting nonverbal synchronization in various human communication situations.

Look at you, look at me: Detection and analysis of mutual gaze convergence in face-to-face interaction

In human communication, mutual attentiveness between individuals is essential for social interaction. The most significant and reliable indicator of mutual attentiveness is gaze. Despite the availability of diverse exorbitant eye-tracking devices, all claim highly accurate gaze positioning, which exceeds the necessity for average researches and applications. We propose an alternative model for mutual gaze convergence detection using economical video capturing devices and straightforward techniques that can investigate natural gaze behaviors in human-human interaction. We conducted a task supposing lecture in face-to-face interaction and applied our model to investigate (1) average percentage of total mutual gaze convergence, (2) duration of mutual gaze convergence, (3) the domination relationship between lectures and students during short lectures, including (4) the effects of attention span, which claimed to last for 10-20 minutes, toward short lecture task (5 minutes). Our results showed that the average total mutual gaze convergence between the lecturer and the student is 52.83%. It reveals the fact that in both long and short lecture, humans have momentary gaze fixation at a particular direction. We also found out that attention span has effect on different durations of lecture task, resulting in different dominance between lecturer and student, be it student for long lecture and lecturer for short lecture. We believe that our developed model implementation and findings exhibit practical applicability and worthy of contribution to the communities of related research areas.

The simultaneous perception of auditory–tactile stimuli in voluntary movement

The simultaneous perception of multimodal information in the environment during voluntary movement is very important for effective reactions to the environment. Previous studies have found that voluntary movement affects the simultaneous perception of auditory and tactile stimuli. However, the results of these experiments are not completely consistent, and the differences may be attributable to methodological differences in the previous studies. In this study, we investigated the effect of voluntary movement on the simultaneous perception of auditory and tactile stimuli using a temporal order judgment task with voluntary movement, involuntary movement, and no movement. To eliminate the potential effect of stimulus predictability and the effect of spatial information associated with large-scale movement in the previous studies, we randomized the interval between the start of movement and the first stimulus, and used small-scale movement. As a result, the point of subjective simultaneity (PSS) during voluntary movement shifted from the tactile stimulus being first during involuntary movement or no movement to the auditory stimulus being first. The just noticeable difference (JND), an indicator of temporal resolution, did not differ across the three conditions. These results indicate that voluntary movement itself affects the PSS in auditory–tactile simultaneous perception, but it does not influence the JND. In the discussion of these results, we suggest that simultaneous perception may be affected by the efference copy.

Body and psychological state synchrony and change by the grant of prior knowledge

According to recent studies, the existence of previous knowledge has been found to be involved in more advanced knowledge acquisition in the human study. Further, when the body motion synchronized, empathy for others has also been found to be important key of interaction. In this study, we focus on those points, by looking at whether the psychological state and body motion are affected by the prior knowledge, and of evaluating the internal understanding and consent of knowledge understanding from the body synchrony will consider the possibility. Specifically, we conducted the experiment lecturer task on teacher-student interaction. There are two group conditions. One is given prior knowledge, the other is not given prior knowledge. The students of the two groups are given same lecture. psychological state is measured using a questionnaire, physical motion is measured using an acceleration sensors. The results reveal that in prior knowledge group, the psychological synchrony is likely to occur more than no prior knowledge group and the body synchrony is found to occur at an early stage with closer to 0 time lag. These differences are occurred due to the involvement of the lecture. The students with prior knowledge is more likely to be involved in the lecture more than the students with no prior knowledge.

Temporal order perception of sensory stimuli under motion perception in audiovisual processing

Motion perception has an important role in a dynamic environment. However, the mechanism of perception of external stimuli under motion perception is not fully clarified. This study investigated the relationship between temporal order perception and motion perception on audiovisual processing. Participants performed audiovisual temporal order judgment (TOJ) task under apparent motion condition and non-apparent motion condition in Experiment 1 and also under random-order presentation condition in Experiment 2. Our result shows that the perceived order and temporal resolution between audiovisual stimuli is different between apparent motion perception and non-apparent motion perception. Besides, the perceived order during apparent motion perception was processed regardless of prediction. In particular, motion perception and exogenous attention are closely related. The relationship shows that motion perception may be exogenously processed by bottom-up signals of external stimulus.

Temporal processing on audiovisual simultaneity during perception of apparent motion

The relation between visual motion information and temporal perception has a significant effect on the development of man-machine interface. However, the relation is still not fully understood. This study aims to investigate temporal processing of audiovisual simultaneity during perception of apparent motion, which is the fundamental unit of human motion perception. Participants performed an audiovisual temporal order judgment (TOJ) task under two conditions: apparent motion condition and non-apparent motion condition. Our result shows that visual motion information contributes to the acceleration of visual processing and the increase of temporal resolution in temporal processing of audiovisual simultaneity. Our findings will provide useful information to construct the frame of temporal processing in man-machine interface.

Visual motion information influences audiovisual simultaneity

Motion information plays an important role in our daily life and temporal information is influential in dynamic environment. We therefore investigated how visual apparent motion information influences temporal perception. Participants performed audiovisual temporal order judgment (TOJ) task under apparent motion condition and non-apparent motion condition. We showed that visual apparent motion has resulted in faster visual processing and higher temporal resolution compared with non-apparent motion. Our findings suggest motion information accelerates temporal perception and improves temporal resolution.