If I had to choose one neurological disorder to be afflicted by...

In 1954, RCA released the first color television set, transforming shades of gray into rich, dazzling hues at dinner tables around the country. Imagine experiencing that tantalizing technicolor transformation when you hear a piece of music, or look at everyday objects like numbers, letters, and days of the week? There is a rare condition in which people do experience the world in this extraordinary way, such as Nobel prize-winning physicist Richard Feynman:

"When I see equations, I see the letters in colors – I don't know why. As I'm talking, I see vague pictures of Bessel functions from Jahnke and Emde's book, with light-tan j's, slightly violet-bluish n's, and dark brown x's flying around. And I wonder what the hell it must look like to the students."

This condition, in which certain sensory stimuli trigger unusual additional sensory experiences, is called "synesthesia." Grapheme-color synesthetes, like Feynman (as well as Vladmir Nabokov), look at printed black letters or numbers and see them in color, each a different hue. For example, 2 might appear dark green, 5 might be red, and 7 may be tinted orange, even though the synesthete is well aware that the numbers are black. Others see or "experience" colors when they hear certain musical tones ("sound-color synesthetes," most famous are Duke Ellington and Wassily Kandinsky); in others, individual words of spoken language evoke vivid taste sensations in the mouth.

This fascinating mingling of the senses was first brought to the attention of the scientific community in 1880, when Francis Galton (cousin of Charles Darwin) described the phenomenon in Nature. He described individuals with grapheme-color and sound-color synesthesia, and proposed that the condition was inheritable (a hypothesis recently supported by work from Simon Baron-Cohen, cousin of Sacha Baron-Cohen).

Galton's work was followed by a brief period of scientific interest in synesthesia, but because the condition could not be observed by anyone but the beholder, it was soon brushed aside as a curious anomaly, presumably the product of insanity, drugs, and/or an overactive imagination. In addition to the questionable neural basis, it was doubtful whether there were any significant implications beyond the phenomenon itself, thus offering little to tempt the scientific community. In the 1990's, however, internal states like consciousness became respectable areas of investigation, and attention returned to synesthesia.


In order to demonstrate that synesthesia was a real phenomenon, researchers designed clever cognitive tests that would reveal their abilities. V.S. Ramachandran, for example, showed that synesthetes could perceptually group graphemes according to their synesthetic colors. He designed a task in which a triangle composed of 2's was embedded in a background of 5's. As you can see in the top box, the numbers are similar enough (they are mirror images of vertical and horizontal lines) that they blend together. In order to discern the shape, one must actively search for the 2's in the sea of 5's. But imagine that the 2's are green and the 5's are red (as in the bottom box). It is now effortless (for most of us) to segregate the two numbers by color, and normal humans instantly perceive the shape. Similarly, a grapheme-color synesthete who perceives 2's and 5's as distinct colors can look at the top array of black digits and effortlessly discern the triangle of 2's.

Another intriguing clinical test for synesthesia, also designed by Ramachandran, takes advantage of the visual phenomenon known as the "crowding effect." If a person is staring straight ahead, and a number (e.g. 5) is presented off to one side, it is easy to discern. However, if the 5 is flanked by other numbers ("distractors," e.g. 3), the average person finds it difficult to recognize the middle number (an effect thought to result from limits in visual attention). Likewise, a synesthete will be unable to discern the middle number, but will still be able to identify it "because it looks red [or whichever color he or she associates with 5]"! Thus, even though the individual is not consciously aware of the number, it still evokes its respective color.

These studies, along with earlier research by Baron-Cohen, have established that syensthesia is clearly a very real sensory/perceptual phenomenon. Understanding the neural basis for this curious interweaving of the senses thus has enticing potential for linking the organization of the brain to perception and sensory experience.

So what is it that differs the brain of a synesthete from my brain, which perceives black numbers and letters as their dreary black selves? What happens to the visual information in the synesthetic brain such that it is transformed in an extraordinary way? In order to begin developing theories of how grapheme-color synesthesia might work, it's important to have an understanding of how the brain processes visual information. (There are, as I mentioned, many types of synesthesia: sound-color, sound-taste, grapheme-taste, texture-taste etc. Grapheme-color synesthetes, however, are the most common subset (representing 68% of all synesthetes), and are the easiest to study, hence this discussion, and most research, is limited to latter condition).

After light reflected from an object hits the cones in the back of the retina, the neural signal travels along several layers of neurons to the retinal ganglion cells. These cells send their axons out the back of the eye, through the optic nerve, to a small part of the thalamus called the lateral geniculate nucleus (LGN), which relays the stimulus directly to the primary visual cortex ("V1", aka "area 17" aka "striate cortex") in the back of the brain. If you locate that bump on the back of your head, V1 is right on the other side of the skull. (There is another major visual pathway--the retinotectal pathway--which bypasses the LGN and V1, but this pathway does not transmit information about color so I'll ignore it in this post). In V1, the visual information is partitioned into visual attributes such as color, form, and motion. Information from these categories is then communicated to the respective processing regions; for color, this is region V4, which is located in the fusiform gyrus of the inferior temporal cortex. After V4, the information is relayed to cognitively "higher" processing centers, including a region of the temporoparietal-occipital (TPO) junction, a structure on which multiple sensory pathways converge.

What about numbers and visual graphemes? Lo! Studies in humans and monkeys have shown that the shapes of numbers and letters are also processed in the fusiform gyrus, in a region adjacent to V4. Moreover, numerical concepts, such as sequence and quantity, are processed in the TPO.

The similarity and proximity of the color and grapheme processing routes has led to the hypothesis that there is some abnormal form of communication occurring between the two in the synesthetic brain. As a result, any time there is an activation of neurons representing numbers, there may be a corresponding activation of color neurons.

This insight has given rise to two neural models for synesthesia. According to one idea, synesthesia results from abnormal connections between the relevant brain areas. During development, the human fetus has dense interconnections between V4 and other inferior temporal regions (, most of which are removed through a process of pruning later in development. Synesthesia may result from a partial failure of this normal pruning process, resulting in excess connections between normally isolated sensory areas. Perceptually, this would lead to a blurring of the boundaries that normally exist between the senses.

The other neural model is called the "disinhibited feedback" theory, which posits that the connections in the brain of a synesthete are no different from those in the normal human adult. Remember that the TPO is a multisensory integration center, receiving information from multiple sensory pathways. This nexus also sends reciprocal feedback connections back to the contributing sensory areas (e.g. the TPO responds to input from V4 by sending output back to V4). This feedback is inhibited when its respective area is not activated (TPO will not send feedback to V4 if V4 has not provided it with input). In synesthetes, this theory proposes, one type of sensory information (e.g. a grapheme) may induce abnormal disinhibition of the feedback pathway of a different sensory pathway (e.g. color), thus propagating its information down the "wrong" pathway to a functionally distinct area. Thus, viewing a particular number may disinhibit the pathway that activates neurons representing a particular color in V4. This hypothesis is supported by accounts of synesthesia being induced by hallucinogenic drugs, implying that the experiences rely on normally existing circuitry, as opposed to the formation of new connections.

Although the cross-activation theory seems to have a bit more support in the scientific community, there was, until now, little evidence demonstrating that one theory was more accurate than the other. This week, however, Romke Rouw and H Steven Scholte of the Univeristy of Amsterdam made an important contribution to the field by examining the structural connectivity of the synesthetic brain. Their results, published online in Nature Neuroscience, demonstrated that the degree of structural connectivity was correlated with the existence and nature of the synesthetic experience.

One of the important methodological issues with this study is the acknowledgment of the heterogeneity of synesthetes (although the study was confined to grapheme-color synesthetes). Synesthetes with the most dramatic experiences of synesthesia actually see colors projected onto letters or numbers, and are referred to as "projectors." The majority of synesthetes, however, do not experience their colors in external space; instead, they use phrases like "in my mind's eye" or "in my head." The colors are just as specific and repeatable as those perceived by projectors, but are sensed internally. Such synesthetes are called "associators."

In line with the "cross-activation" theory explained above, the researchers explored whether there were, in fact, more connections in synesthetes than non-synesthetes, focusing their analysis on the fusiform gyrus. To examine the neural connectivity, the researchers used a technique called diffusion tensor imaging (DTI), which measures the direction of movement of water molecules. In most brain tissue, water molecules diffuse chaotically, at random. Along the myelin sheaths of axons, however, water movement is restricted, thus following the path of the axon. This technique allows the visualization of bundles of axons; more (or more densely bundled) connections will yield a higher signal. Thus, the strength of the DTI signal is related to the strength of the connection.

Their results confirmed that the brains of synesthetes have increased connectivity in the inferior parietal cortex (near the fusiform gyrus), as well as the frontal and parietal cortices (involved in controlling spatial attention) than in normal individuals. Morevover, subjects with the strongest connectivity at the fusiform gyrus were projectors, while associators had connections stronger than controls but weaker than the former. This hyperconnectivity is thus tightly correlated with the synesthetic experience, offering a neural basis for this sensory fusion.

The study also found significantly higher levels of connectivity in the frontal and parietal cortices of synesthetes, with no difference between associators and projectors. These areas had not been previously linked to synesthesia, but the authors mention that they may be involved in perceptual transitions, such as the fluctuations that occur when viewing bistable figures like Rubin's face-vase figure (on the right) or during binocular rivalry. This may be related to the synesthetic experience; many synesthetes see the color only transiently or as a flickering perception.

This study is the first to demonstrate that increased connectivity in specific areas of the brain is related to synesthesia. It is certainly possible that this structural phenomenon is supplemented by abnormal disinhibited feedback, or that it accounts for only a subset of synesthetic cases, and more studies are needed to support these theories. Moreover, it will be interesting to see whether similar structural abnormalities are present in cross-modal synesthesia, such as sound-color or sound-taste, which sensory centers are more isolated than the adjacent color and grapheme perceptual centers.

References:
Feynman, Richard (1988). What Do You Care What Other People Think? New York: Norton. P. 59.Rouw R and Scholte HS. Increased structural connectivity in grapheme-color synesthesia. Nature Neuroscience [Published online May 21, 2007]

Color me fantastic

For some background on vision, particularly color vision, read my previous post.

Millions of years ago, before the dinosaurs roamed, the remote ancestors of mammals probably had magnificent color vision. When the dinosaurs came to ecological power, these ancestors (mammal-like reptiles), were banished from the daylight hours and became creatures of the night. During this period, their visual systems evolved be maximally sensitive to light, allowing their proficiency at color discrimination to deteriorate in exchange. Although many mammals have returned to reclaim their diurnal dominion, their color vision still pales (get it?) in comparison to that of other vertebrates, such as reptiles, fish, and birds, whose ancestors did not suffer this period of exile.

Most mammals (some primates, including humans, are exceptions) rely on a dichromatic visual system, with cones that respond optimally to either short (S) or medium (M) wavelengths. (For more on this, see the primer) Our remote ancestors, along with the aforementioned other vertebrates, had three, four, or possibly more types of cones. A broader range of cones enables an exponentially enhanced ability to distinguish colors, but was unnecessary during our ancestors' tenure as nocturnal animals and was subsequently lost.

Our more proximal ancestors, apes and Old World (African) monkeys, regained a third cone, the long (L) wavelength-responsive cone, an adaptation that was instrumental to our evolutionary success. This cone allowed primates to distinguish red from green, which would have been advantageous for purposes of foraging. There are two major questions that arise from this trichromatic revival: How did it happen? and How did our higher visual systems adapt to interpret/perceive this new dimension of sensory experience?

The first is a question of genetics, which I'll answer briefly before delving into the neuroscience. Outside of the lab, new genes aren't added to the genome de novo; they arise from duplications of extant genes. Over time (on an evolutionary timescale), mutations change one or both copies of the gene such that they end up coding for different proteins. The genes that code for the M and L cones are very similar, and are on the same chromosome; it is thus likely that when Old World monkeys regained trichromacy, the gene for the M cone duplicated and mutated to form a gene that coded for the L cone (the divergence of color sensitivities would have been favored by natural selection). It is further believed that the M cone originated from a duplication/mutation/diversion combo from the S cone (which is on a different chromosome).

So that's how it happened on a genetic/peripheral level, but what about the mind? What good is it to be able to have this increased diversity in the retina if the brain is only capable of processing a dichromatic visual world? How much time had to pass before we could not only see this new dimension of light, but could interpret it as well?

A very exciting paper came out recently in Science, in which a group led by Gerald Jacobs at UCSB (the king of color vision, I'm told) used genetically engineered mice to model this crucial adaptation to trichromacy. Mice are, like most mammals and our ancestors, dichromatic, with only S and M cones. The group furnished the mice with the gene for the human L cone, resulting in mice with all the retinal components of trichromatic vision. They then sought to determine whether the mouse nervous system would be able to capitalize on this new information, thus endowing the mouse with an enhanced sensory experience. Alternatively, the visual system may be genetically wired such that it could only handle inputs from the two existing classes of cones, implying that the homologous primate adaptation would have required generations for adaptive rewiring.

To address this question, the group wanted to see if the "trichromatic" mice were able to distinguish between two colors which their dichromatic siblings (and ancestors) would have considered the same. To do this, they put them in front of three colored panels, on which two different wavelengths or intensities were displayed (they should be able to distinguish, for example, a dim green light from a bright red light...for background see primer). The mice were trained to identify which of three panels was illuminated with a different color than the other two; a correct choice was rewarded with drops of soymilk. After about 17,000 tests (literally), most of the mice with the S, M, and human L cone were able to learn the task, and chose correctly 80% of the time! (Mice with only the S and M cones still performed no better than chance after a similar number of trials.)

This demonstrates, quite wonderfully, that "the mammalian brain is sufficiently plastic that it can extract and compare a new dimension of sensory input." It follows, they speculate, that the primate that first inherited an L cone would have immediately reaped the benefits, and would likely have enjoyed a selective advantage.

I think this is quite a marvelous find, and the paper made me happy and excited, but I am also not entirely surprised by the result. It is merely a reminder of the essence of evolution, in that it tends to be guided by everyday biological mechanisms; things that seem special or mysterious in our natural history and our living world are made possible by these same mechanisms. I've previously discussed this idea that the parsimony of evolution would allow novel adaptations in the periphery to exploit previously evolved central circuitry. For this study, I think it's helpful to supplement this idea with a brief discussion of the striking plasticity of the developing nervous system.

When an animal is developing, the pathways of the visual system are highly sensitive to certain visual stimuli, and grow and refine under the guidance of the animal's experience. Some of you may have heard of experiments in which kittens were raised in boxes in which their only visual stimuli were black and white vertical stripes. Their visual pathways developed such that neurons in the visual processing areas responded primarily to vertical lines, and failed to respond to horizontal lines when presented later in life. In a sense, their brains were not able to see horizontal planes.

It seems to follow that if an animal is endowed with a "new" element of visual stimulation, rather than deprived, its developing visual pathways would accommodate this visual world by wiring up accordingly. As the brain of a dichromat was already capable of comparing information from two different classes of cones to compute a color representation in the brain, it may not have been too demanding a request to factor a third into the equation. Importantly, this adaptation would take place during the development of the first animal that possessed the new cone, as opposed to an evolutionary timescale.

This brings up an intriguing thought: what would happen if a mutation arose that endowed humans with a fourth cone? Our resulting percept of the world is a bit difficult to fathom...would we be able to distinguish light mixtures that normal people cannot? What would these colors look like? What if this new cone was able to detect light outside our visible spectrum--ultraviolet light, like bees and some birds, or infrared light, like rattlesnakes?

Vision primer

My trichromatic mouse post was way too long, so I decided to take all the background information on the visual system, which some of you may not need, and put it in a different post, thus breaking it into more digestible bits. This intro focuses on topics relevant to the article (photoreceptors and the initial processing stages of color vision), so although I would love to discuss physical nature of light, the complexity of the vertebrate eye, and higher visual processing, I'm leaving all that for another day!

Vision
Our visual system exploits light to generate a useful representation of our environment. This process involves a highly specialized division of labor, in which our eyes function as the light "detectors." They then convey this data in an organized fashion to the visual areas of the brain, which process this deluge of information such that we can make sense of our visual world.

This sounds basic, but I want to make sure I preclude the common misconception that our eyes are like tiny cameras, which project tiny inverted images of the world for the brain to "watch" like a movie on a screen. Instead, when light (with wavelengths between 700 billionths of a meter to 420 billionths of a meter) hits our eyes, it activates a certain pattern of sensory neurons in the retina (photoreceptors), and after some preliminary organization at the level of the retina, a modified pattern is conveyed to the brain, where it is decoded in the brain by a mechanism which is far from understood. So not only do our photoreceptors not relay "picture of a tree," they do not even convey information such as "green."

Color Vision
How, then, do we see "green?" Our eyes rely on a three-color system ("trichromacy"); that is, they contain three distinct classes of color photoreceptors (called cones), each of which respond differently to a given wavelength and intensity of light. The classes are generally called short-wave (S), medium-wave (M), and long-wave (L), in reference to their optimal responsiveness to either short, medium or longer waves of light. They are more commonly called Blue, Green, and Red cones, colors which correspond to their respective wavelength (although Red is somewhat a misnomer because the neurons actually respond optimally to Yellow).

However, these cones are pretty broadly tuned, meaning they are actually capable of responding to wide ranges of light; the functional difference between an optimal wavelength and a suboptimal wavelength for a given class is a difference in the firing rate of the cone. To add further confusion, the firing rates are dependent not only on the wavelength, but also to the amount (intensity) of light.

This explains why the magnitude of response of a single class of cones cannot tell us anything about the color of a light: a "green" cone will respond to a dim green light with the same firing rate as it will to a bright red light. In order to deduce the color of an object, it is thus necessary for the brain to compare the responses of representatives from each class of cones. With trichromatic vision, our brains can compare the firing rate of three classes of cells, all with different properties, giving us the ability to perceive and differentiate among a dazzling array of hues.

Most mammals have dichromatic vision, with only two cones (short and medium); seeing through their eyes would be akin to missing a cone, which is the case with the 8% of men and 0.5% of women who are colorblind. Many fish and reptiles, on the other hand, are tetrachromatic, and the retinas of birds and turtles may exhibit even more diversity.

To see what happens when you give a dichromatic animal trichromatic vision, read on...

What do dodgeball and ventriloquism have in common?

Imagine a game of dodgeball, in which your attention is consumed by whizzing balls and the warlike cries of aggressive athletes, all reverberating off the gymnasium walls. Suddenly, you hear someone on the opposing side yell to his teammates that his next shot will be aimed at you. At that instant, you look over at his side of the court and see three hostile-looking boys holding dodgeballs, approaching the halfline, mouths moving as they boldly communicate strategy and support. Which one identified you as his target? Luckily, the same voice continues its warmongering, designating targets for his other teammates, and you are now able to match the movement of the mouth of one of the boys with the relevant sounds. Sound and sight have come together, and you are prepared for the appropriate ball.

Amidst that bombardment of sensory information, how did your brain find the appropriate auditory-visual correspondence to determine the origin of the battle cry? At every moment, in dodgeball and in life, our appreciation of the external world is due to a combination of sights, smells, sounds, touches, and tastes. Our brains must integrate this deluge of information to generate a coherent, seamless picture of the environment; this process is called multisensory integration. When integrated properly, the simultaneous acquisition of information from different sources helps us refine our percept of the world. However, our incoming sensory information is often fraught with uncertainty.

To explore this concept on sensory uncertainty, it is useful to focus on one sensory modality, such as vision. Getting back to the game: we know where the ball is coming from, but we still need to dodge (or catch!) it. Your assailant winds his bulging arm back like a catapult and mechanically releases the projectile; as it careens towards you, you attempt to calculate its speed and trajectory. Your eyes convey imperfect information about the ball's velocity, so your brain can only estimate it. Combining this information with your memory of his previous throws reduces the error in this estimate, but not all velocities are equally probable in theory; over the course of the game, there will be a probability distribution of velocities. Your best estimate, and your ability to dodge the ball, results from combining information about the distribution of prior velocities with evidence from sensory (visual) feedback. This interpretation of probabilities is called "Bayesian inference," and various studies have shown that the human brain performs Bayesian inference at a nearly optimal level.

Multisensory integration becomes far more complex when we consider this uncertainty inherent to our sensory information. Nevertheless, our brains are intriguingly capable of weighing different sensory signals according to their corresponding reliabilities; that is, our brains pay more attention to "reliable" sensory information, while disregarding "unreliable" information. This ability results in an "optimal" approximation of reality: we are (nearly) perfect maximum-likelihood integrators.

The complexity of this integration process is exposed when our perceptual world does not correspond to reality. One striking example of this vulnerability is ventriloquism. A good ventriloquist will thwart our multisensory integration process by synchronizing the movements of a puppet's mouth with his or her voice, while the movements of his or her mouth are imperceptible. Thus, we perceive the voice as originating from the puppet, as opposed to the ventriloquist.

This deception is a consequence of our brain's propensity to give more weight to visual information than auditory information during the integration process; the neural circuits have adapted to the fact that the visual system is far more reliable at determining location than the auditory system. The direction of a light source is directly determined by the position stimulated on the retina, whereas the direction of a sound is calculated by the differences in timing and intensity of stimulation in one ear relative to the other. Thus, our brain "trusts" the visual system more than the auditory system, and rightly so. If there is a discrepancy between the two, the visual information is favored in the generation of a unified percept of reality, and the puppet "speaks."

So, the big question is, as always: what are the neural mechanisms underlying this process? How does the brain weigh different signals according to their corresponding reliabilities when generating the most realistic percept? How is uncertainty represented at the neural level?

A group in Alex Pouget's laboratory recently published theoretical answers to these questions in Nature Neuroscience. The premise for their exploration was the fact that neurons in the cortex respond to identical stimuli with high variability. For example, think of a neuron in the visual cortex that responds to an object moving from left to right. When exposed to such a stimulus, it will not respond exactly the same way each time: the same neuron may respond by firing 9 times, or 14 times, or not at all. Although this particular cell is, on average, activated by left-to-right motion, its response to this stimulus may change dramatically from one presentation to the next.

Pouget's group hypothesized that this variability may represent sensory uncertainty. Let’s return our focus to the visual system and dodgeball. Your uncertainty of the speed at which the dodgeball is moving is related to the fact that neurons in your visual cortex do not fire in exactly the same way every time you see a ball moving towards you. Balls flying at your head can look different depending on your vantage point (and other factors, such as the physical properties of the ball itself), and thus give rise to different responses in your visual cortex every time. If an approaching dodgeball always elicited the same neural responses, you would be able to determine its speed with certainty, by evoking your memory of the ball's speed in past occasions.

The researchers showed mathematically that this variability could represent probability distributions for an object's location. Greater uncertainty (i.e. wider probability distributions) would thus be represented by higher variability in responses of neurons in the auditory cortex relative to those in the visual cortex. This internal representation of sensory uncertainty allows the brain a relatively straightforward (linear) way to combine neural activities: the Bayesian “decoder” of the brain can simply pool the probability functions of multiple neurons (which can represent multiple sensory channels) together to generate an optimal inference of an object's location.

So when watching a ventriloquist act, our visual system detects the movement of a puppet’s mouth, which neurons in our visual cortex register with low variability (high certainty and a narrower, more mathematically dominant probability function), while the neurons in our auditory cortex represent sound originating from the mouth of the ventriloquist with high variability (low certainty and a wider, less influential probability function). When the brain combines these functions together with its Bayesian decoder, the visual system "wins" and we think the sound came from the puppet.