"When I see equations, I see the letters in colors – I don't know why. As I'm talking, I see vague pictures of Bessel functions from Jahnke and Emde's book, with light-tan j's, slightly violet-bluish n's, and dark brown x's flying around. And I wonder what the hell it must look like to the students."
This condition, in which certain sensory stimuli trigger unusual additional sensory experiences, is called "synesthesia." Grapheme-color synesthetes, like Feynman (as well as Vladmir Nabokov), look at printed black letters or numbers and see them in color, each a different hue. For example, 2 might appear dark green, 5 might be red, and 7 may be tinted orange, even though the synesthete is well aware that the numbers are black. Others see or "experience" colors when they hear certain musical tones ("sound-color synesthetes," most famous are Duke Ellington and Wassily Kandinsky); in others, individual words of spoken language evoke vivid taste sensations in the mouth.
This fascinating mingling of the senses was first brought to the attention of the scientific community in 1880, when Francis Galton (cousin of Charles Darwin) described the phenomenon in Nature. He described individuals with grapheme-color and sound-color synesthesia, and proposed that the condition was inheritable (a hypothesis recently supported by work from Simon Baron-Cohen, cousin of Sacha Baron-Cohen).
Galton's work was followed by a brief period of scientific interest in synesthesia, but because the condition could not be observed by anyone but the beholder, it was soon brushed aside as a curious anomaly, presumably the product of insanity, drugs, and/or an overactive imagination. In addition to the questionable neural basis, it was doubtful whether there were any significant implications beyond the phenomenon itself, thus offering little to tempt the scientific community. In the 1990's, however, internal states like consciousness became respectable areas of investigation, and attention returned to synesthesia.
In order to demonstrate that synesthesia was a real phenomenon, researchers designed clever cognitive tests that would reveal their abilities. V.S. Ramachandran, for example, showed that synesthetes could perceptually group graphemes according to their synesthetic colors. He designed a task in which a triangle composed of 2's was embedded in a background of 5's. As you can see in the top box, the numbers are similar enough (they are mirror images of vertical and horizontal lines) that they blend together. In order to discern the shape, one must actively search for the 2's in the sea of 5's. But imagine that the 2's are green and the 5's are red (as in the bottom box). It is now effortless (for most of us) to segregate the two numbers by color, and normal humans instantly perceive the shape. Similarly, a grapheme-color synesthete who perceives 2's and 5's as distinct colors can look at the top array of black digits and effortlessly discern the triangle of 2's.
Another intriguing clinical test for synesthesia, also designed by Ramachandran, takes advantage of the visual phenomenon known as the "crowding effect." If a person is staring straight ahead, and a number (e.g. 5) is presented off to one side, it is easy to discern. However, if the 5 is flanked by other numbers ("distractors," e.g. 3), the average person finds it difficult to recognize the middle number (an effect thought to result from limits in visual attention). Likewise, a synesthete will be unable to discern the middle number, but will still be able to identify it "because it looks red [or whichever color he or she associates with 5]"! Thus, even though the individual is not consciously aware of the number, it still evokes its respective color.
These studies, along with earlier research by Baron-Cohen, have established that syensthesia is clearly a very real sensory/perceptual phenomenon. Understanding the neural basis for this curious interweaving of the senses thus has enticing potential for linking the organization of the brain to perception and sensory experience.
So what is it that differs the brain of a synesthete from my brain, which perceives black numbers and letters as their dreary black selves? What happens to the visual information in the synesthetic brain such that it is transformed in an extraordinary way? In order to begin developing theories of how grapheme-color synesthesia might work, it's important to have an understanding of how the brain processes visual information. (There are, as I mentioned, many types of synesthesia: sound-color, sound-taste, grapheme-taste, texture-taste etc. Grapheme-color synesthetes, however, are the most common subset (representing 68% of all synesthetes), and are the easiest to study, hence this discussion, and most research, is limited to latter condition).
What about numbers and visual graphemes? Lo! Studies in humans and monkeys have shown that the shapes of numbers and letters are also processed in the fusiform gyrus, in a region adjacent to V4. Moreover, numerical concepts, such as sequence and quantity, are processed in the TPO.
This insight has given rise to two neural models for synesthesia. According to one idea, synesthesia results from abnormal connections between the relevant brain areas. During development, the human fetus has dense interconnections between V4 and other inferior temporal regions (, most of which are removed through a process of pruning later in development. Synesthesia may result from a partial failure of this normal pruning process, resulting in excess connections between normally isolated sensory areas. Perceptually, this would lead to a blurring of the boundaries that normally exist between the senses.
This study is the first to demonstrate that increased connectivity in specific areas of the brain is related to synesthesia. It is certainly possible that this structural phenomenon is supplemented by abnormal disinhibited feedback, or that it accounts for only a subset of synesthetic cases, and more studies are needed to support these theories. Moreover, it will be interesting to see whether similar structural abnormalities are present in cross-modal synesthesia, such as sound-color or sound-taste, which sensory centers are more isolated than the adjacent color and grapheme perceptual centers.
Feynman, Richard (1988). What Do You Care What Other People Think? New York: Norton. P. 59.Rouw R and Scholte HS. Increased structural connectivity in grapheme-color synesthesia. Nature Neuroscience [Published online May 21, 2007]