How to See a Sound

(1) Imagine striking this object with a felted mallet.

 

How does it sound? You can probably hear it in your head. What if you struck it with an iron hammer? Ditto. Does this imaginative success mean that you see its sound?

Some think so. More than sixty years ago, Roderick Firth wrote a deep and nuanced paper in Mind contesting what he called “Sensory Core Theory,” according to which perceptual appearance consists of sense-data interpreted in the light of “custom and habit.” When you see the bell, you receive visual sense data. These are experienced not merely as sense-data, but as a bell. Thus, you see a bell. Auditory sense data are different from visual ones and belong to a different experience. You may hear, but you don't see, the sound of the bell.

Firth urged a different view, which he called Percept Theory, according to which “all those things that are present to consciousness in perception are present in exactly the same way.”—“a ripe tomato hanging on a vine in the sun is ‘leibhaft gegeben’ with all its sensuous qualities of redness, and smoothness, and warmth, and sweetness.” According to Firth, then, there is no phenomenological distinction between seeing, supposedly directly, that the bell is metallic grey and seeing indirectly that it makes a certain sound when struck.

(2) I do not think that Firth is right about this, but this isn't what I want to discuss here. I want instead to raise a related question. When you witness two cars collide, or when somebody in front of you is speaking, do you see the sounds? This may strike you as the same question, but it is actually a special case.

Notice first that the associationist "custom and habit" view does not apply to this case. For it is a fundamental principle of the theory of conditioning that what is learned by association can be unlearned. If the bell was doctored to make a different kind of sound when hit by a mallet, you’d begin after a while to hear that different sound in your imagination. But this does not happen when you see sound-events such as those mentioned just now. In the Ventriloquist Effect, the dummy’s lips are flapping, but the ventriloquist produces the sound. You hear the voice coming from the dummy’s lips: you cannot override the illusion, no matter how many times you experience it. Similarly, if you see a display in which two discs cross silently, you will see them trace the arms of an X; if a click is provided at the time of intersection, you will see the two discs bouncing off each other and reversing their direction. The two displays are optically identical but very different from the phenomenological point of view. Your perceptual system locates the sound where vision sees it being made. Does this not imply that vision sees the sound?

You may resist. You concede that one sees the motion of the dummy’s lips. But you insist that one does not see it making sound. But this response has been contested in a brilliant 1999 paper by Robert Pasnau in Philosophical Quarterly. Pasnau argues:

1. You hear things as located in space.
2. Auditory perception is accurate when the location of what is heard is that of the event that caused you to hear what you did.
3. So: what you hear is the event that causes you to have the auditory experience.

The flapping of the dummy’s lips is the sound. And you see it.

Pasnau’s insight deflects another response. You may think that what you see is a causal connection. And you might draw on the work of Albert Michotte to support your hunch: causation can be visually perceived. Accordingly, you might say: you see the flapping of the lips causing the sound. This line of argument faces two major difficulties. The first has already been mentioned: Pasnau’s insight suggests that you are making a distinction without a difference. But secondly, Michotte’s work was about sequences in which there is a small but significant temporal lag between “cause” and “effect”. In this case, there is no such time lag. Moreover, Michotte’s events were all visual. It has yet to be shown that causality can be perceptually registered in multisensory sequences.

(3) An intriguing question. Do you see sounds when you read? You take in visual information; you hear it in your head as sound; you have a sensorimotor response (subvocalization) matched to the sound in your head; the experience gives you reason to believe that a certain text is before you.

I don’t really know what to think about this one. Here are some relevant questions. Are texts auditory objects? When you read, do you come in contact with such an auditory object? Or are texts abstract objects that have auditory realizations/instances? And when you read do you come in contact only with the abstract object, not its auditory realization? Is a “Platonic” theory of reading right? That is: when you read do you come in contact with a visual phenomenon that “reminds” you of an abstract object, a text? And when in contact with this abstract object are you then put in mind of its auditory realization?

The ontology of sound has been fruitfully explored by Pasnau, Casey O’Callaghan, and Matthew Nudds. (I have made some modest contributions as well.) The ontology of reading awaits philosophical attention.