A Caltech Library Service

Crossmodal Interaction in Humans


Watanabe, Katsumi (2001) Crossmodal Interaction in Humans. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/66cn-aq26.


Since different senses are attuned to the different aspects of the environment, crossmodal interactions can reduce perceptual ambiguity which may result from relying on a single sensory modality. While many studies have shown that visual information affects perception in the other modalities, little is known about how auditory and tactile information affect visual perception. The present thesis explores how the human perceptual system synthesizes visual event perception by using an ambiguous visual motion display: Two identical visual targets moving across each other can be perceived either to bounce off or to stream through each other. A brief sound at the moment the targets coincide biases perception toward bouncing. Extensive psychophysical investigations on this bounce-inducing effect show that: (1) A salient sensory transient biases visual perception toward bouncing, irrespective of the modality in which such a transient is presented. (2) The magnitude of the bounce-inducing effect depends on both the saliency of a sensory transient and the timing of a sensory transient relative to the visual coincidence. (3) The magnitude of the bounce-inducing effect increases as the sensory saliency of bounce-inducing transients increases. (4) The interaction window during which a sensory transient can bias visual perception is fixed in time. (5) Auditory and tactile transients have much larger temporal interaction windows than visual transients. (6) Auditory and tactile transients presented before the visual coincidence tend to have a stronger bounce-inducing effect than those presented after the visual coincidence; visual transients do not show this asymmetry. (7) The lack of attention to the visual moving targets in the streaming/bouncing motion display increases the likelihood of the bouncing percept. Based on these results, I conclude that the streaming percept arises when amodal attentional resource is available for moving stimuli around the moment of the visual coincidence and that the bouncing percept results from the lack of attentional resource. The role of salient sensory transients in the bounce-inducing effect is to distract attention from the coinciding moving stimuli. Thus, this thesis shows that dynamics of attentional resource allocation, not simply associative learning, can be the cause of amodal event perception in humans.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:Computation and Neural Systems
Degree Grantor:California Institute of Technology
Division:Engineering and Applied Science
Major Option:Computation and Neural Systems
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Shimojo, Shinsuke
Thesis Committee:
  • Koch, Christof (chair)
  • Shimojo, Shinsuke
  • Allman, John Morgan
  • Andersen, Richard A.
  • Konishi, Masakazu
Defense Date:1 January 2001
Record Number:CaltechTHESIS:10122010-090303102
Persistent URL:
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:6131
Deposited On:12 Oct 2010 16:44
Last Modified:30 Aug 2022 23:51

Thesis Files

PDF - Final Version
See Usage Policy.


Repository Staff Only: item control page