Inside the Mind of a Synaesthete

"The Figure 5 in Gold," by Charles Demuth, 1928

The Figure 5 in Gold, Charles Demuth, 1928

“I have this rather freakish gift of seeing letters in color,” novelist Vladimir Nabokov told a BBC interviewer in 1962. “It’s called color hearing. Perhaps one in a thousand has that.”

The Russian-born author of Lolita, Pale Fire, and other exuberantly witty books claimed that when he was a child, he saw the number 5 as red, and that he had continued to perceive numbers and letters as having their own distinctive hues.  The interviewer asked Nabokov how the initials of his own name appeared to him. He replied:

V is a kind of pale, transparent pink: I think it’s called, technically, quartz pink: this is one of the closest colors that I can connect with the V. And the N, on the other hand, is  a greyish-yellowish oatmeal color. But a funny thing happens: my wife has this gift of seeing letters in color, too, but her colors are completely different.  There are, perhaps, two or three letters where we coincide, but otherwise the colors are quite different.

It turned out, we discovered one day, that my son, who was a little boy at the time — I think he was 10 or 11 — sees letters in colors, too.  Quite naturally he would say, “Oh, this isn’t that color,  this is this color,” and so on. Then we asked him to list his colors and we discovered that in one case, one letter which he sees as purple, or perhaps mauve, is pink to me and blue to my wife. This is the letter M. So the combination of pink and blue makes lilac in his case. Which is as if genes were painting in aquarelle.

Nabokov’s “color hearing” — a curious neurological phenomenon known as grapheme-color synesthesia — also found its way into his books. In Bend Sinister, the protagonist, Adam Krug, says that the word “loyalty” reminds him of “a golden fork lying in the sun on a smooth spread of pale yellow silk.”

In his autobiography, Speak, Memory, Nabokov launches into a virtuoso synaesthetic reverie: “The long a of the English alphabet… has for me the tint of weathered wood, but a French a evokes polished ebony. This black group also includes hard g (vulcanized rubber) and r (a sooty rag being ripped). Oatmeal n, noodle-limp l, and the ivory-backed hand-mirror of o take care of the white… Passing on to the blue group, there is steely x, thundercloud z and huckleberry h. Since a subtle interaction exists between sound and shape, I see q as browner than k, while s is not the light blue of c, but a curious mixture of azure and mother-of-pearl.”

One of the ravishing pleasures of reading Nabokov is sensing a deep rightness in his word choices (even in English, which was his second language) that goes beyond having a knack for finding le mot juste to make his prose cohere at every level: phonetic, orthographic, and semiotic. Surely the atypical wiring of his brain gave Nabokov an advantage in his quest for this comprehensive unity.

Synaesthetic alphabet

Synaesthetic alphabet

Few writers have mapped this uncanny phenomenon with such obsessive precision, but the gift of multiplex senses turns out to be not as freakish and rare as Nabokov believed. Researchers have learned that even chimpanzees associate low notes with darker colors with high notes with brighter ones. In a recent paper in Psychological Science, David Eagelman of the Baylor College of Medicine argues [PDF link] that synaesthetic ability falls along a spectrum, the end product of multiple processes of neuronal excitation, inhibition, and pruning in the brain gone awry, “all of which happen to converge on the similar result of unusual perceptual or cognitive pairings.”

Another recent study raises the possibility that we are all born with a capacity for sensory crosstalk that diminishes as we get older and our neural networks are streamlined for greater efficiency. Nabokov also believed that we are all born synaesthetes, but because science hadn’t yet elaborated the concept of neuroplasticity, he blamed the loss of the gift on “stupid parents” telling their kids, “It’s all nonsense. An A isn’t black, a B isn’t brown. Don’t be absurd.”

In fact, however, hacking the firewalls between senses may turn out to be a useful skill that parents could teach kids who were not born that way. In an ingeniously designed blog post, Macquarie University autism researcher Jon Brock discussed a recent report by V.S. Ramachandran in Neurocase [PDF link] of a young man with Asperger’s syndrome who was instructed to associate emotions with colors to improve his social perspicacity. As he got older, the young man learned to gauge how he felt about a person by the color of the “halo” around the face.

Nabokov was not alone in his conscious employment of his gift to serve his art. Master painter and digital artist David Hockney relies on his synaesthesic abilities to generate hyper-vivid images that glow with an almost child-like visual innocence. He told author David Burton that when he was designing a set featuring the image of a tree for a production at the Metropolitan Opera of a piece by Maurice Ravel, he listened to the relevant section of the score and “the tree painted itself.”

Felled Trees by David Hockney

Felled Trees by David Hockney

Likewise, the genre-stretching jazz, folk-rock, and avant-garde music of Duke Ellington, Syd Barrett, Alexander Scriabin, and Oliver Messaien [PDF link] was allegedly energized by each composers’ bimodal perceptions. In his epic multi-volume manifesto Traité de rythme, de couleur, et d’ornithologie (“Treatise on rhythm, color and bird song”), Messiaen described chords as “blue-violet rocks, speckled with little grey cubes, cobalt blue, deep Prussian blue, highlighted by a bit of violet-purple, gold, red, ruby, and stars of mauve, black and white.”

For a drearily mono-sensory person like me, it’s tough to read these accounts without feeling a (sour-apple green?) twinge of envy. What would it be like to live in a world in which each prime number generated its own harmonics, every letter of the alphabet was associated with a characteristic odor (the freshly laundered scent of L, the fulsome perfume of Q), and Miles Davis’ “Flamenco Sketches” (on Kind of Blue, natch) shimmered like an iridescent watercolor over the heads of the bored baristas at the local coffee shop?

Color music notation

Color music notation

Now a prolific multimedia artist and writer named Perry Hall [Flash required], who was born with his own version of Nabokov’s quirky gift, has developed an iPhone/iPad app called Sonified that enables even those low on the synaesthetic spectrum to experience light, colors, and movement morphing into sounds.

I first became interested in Hall’s work seeing a series of haunting HD videos made in 2006 that he called Material Study, featuring light dancing on the surfaces of ferrofluids that surge and swell like some kind of protean lava. While convalescing from a bout of Lyme disease, Hall decided that he needed to set his synaesthesia loose in the wild, as he puts it. He and his digital collaborators developed software that siphons the luminance and color values from the video cameras in iPhones and iPads (only later-generation devices like the iPhone 4, 4S and iPad 2 will work correctly) and uses them to trigger stereo samples from a library of CD-quality audio composed for the purpose.

Photo of Perry Hall

Perry Hall, artist, synaesthete, and designer of Sonified

When Hall — who helped create the lush “painted world” sequence in What Dreams May Come, the 1998 film starring Robin Williams — told me about Sonified in email, I knew I had to try it myself. After downloading it from the App Store, I boarded a streetcar here in San Francisco, slipped on a pair of headphones, and aimed my phone out the window just as the train streaked past a row of brightly painted Victorian houses, accelerating through shafts of sunlight and shade on its way into a tunnel.

The effect of the audio-visual-kinesthetic link-up was unexpectedly profound. Instead of feeling like Sonified was imposing its digital soundtrack on the world, I felt I was accessing a layer of reality that is normally hidden from us. It was like a little dose of Morpheus’ red pill in The Matrix.

Sonified screen shot

Sonified screen shot

To give potential users a foretaste of the experience, Hall has uploaded videos to YouTube here, here, here, and here. But experiencing Sonified second-hand rather misses the point. The thrill of using the app is having it respond to optical nuances in real time as you move through spaces that come alive in new and surprising ways. Hall’s ethereal sonic palette may be a bit New Agey for some tastes, but the software offers a teasing glimpse of how much more we could be doing with these powerful multimedia platforms in our pockets. Sonified takes the often mundanely-applied concept of augmented reality (we were promised Terminator vision and got Plaxo QR codes instead) a step closer to the radical departure from sensory business-as-usual that 18th century multimedia pioneer William Blake described in The Marriage of Heaven and Hell:

How do you know but ev’ry Bird that cuts the airy way,
Is an immense world of delight, clos’d by your senses five?

Glad Day or the Dance of Albion, William Blake, circa 1794

Glad Day or the Dance of Albion, William Blake, circa 1794

I asked Hall, who lives in western Massachusetts, what it’s like to walk around every day with naturally augmented senses.

Steve Silberman: When did you figure out that you experience the world differently from most people?

Perry Hall: I’ve been experiencing two different kinds of synaesthesia since I was about 10 years old. The first is “colored hearing,” where sounds become very visual; and the other is a milder type of synaesthesia that is more directly connected to Sonified, in which images, color, and spaces like landscapes inspire tones and sounds. Both types are very emotional for me, and define a profound beauty that compels me to write music or create visual works of art that are reflections of it.

When walking through certain kinds of terrain — usually those involving vast spaces or interactions of light and color — these tones will become present. It’s usually in a natural environment, such as a beach, landscape, or other place where deep space and light can play, stretch, and unfold with a kind of purity. But it’s always there, at least in the background.  I was just in Wales taking a train along the Irish Sea, the landscapes that Turner captured in his paintings. I was filled with a lot of sounds as I moved through this environment.

Wreckers Coast of Northumberland, Joseph Mallord William Turner, circa 1836

Wreckers Coast of Northumberland, Joseph Mallord William Turner, circa 1836

I feel an intense identification with these spaces, as if the distance and distinction between myself and the environment is collapsing, and I identify with the sky or landscape and lose myself. The sounds that come from this environment blur the distinction between seeing and hearing into a single point. Visual dynamics create sonic dynamics — the two are joined, like a mirror, with visual qualities on one side and sonic qualities on the other. When the light slowly fades, the sounds fade; as the light shifts, increases, or decreases, the sounds shift equally. A circuit is made between seeing and hearing, but it’s not seeing and hearing, it’s see-hearing, seering, hearseeing — something distinct from merely a combination of the two senses that we think of as separate.

Silberman: What inspired you to try to make these experiences available to others?

Hall: Eighteen months ago, I was diagnosed with Stage 2 Lyme disease. I had it for over a year, including a fever that lasted for four months. Finding myself in bed, wondering what was going to happen, I had time to think. I realized it was important for me to make this strange and wonderful experience accessible to other people. I came up with a means of creating a “synaesthetic engine” that could translate light and color into sound. I wanted to put this engine into a camera anyone could use anywhere, rather than putting it in a gallery or museum. I wanted people to be able to walk through a desert or a city, to lay in bed or sit on a train, and have the same kind of experiences I can have anywhere. Knowing I was stuck in bed, a friend gave me an iPhone to play around with, and I saw it had all the pieces that I needed to set synaesthesia loose in the wild.

Silberman: How did you go about designing the software?

Hall: First I created the design documents — a set of charts showing one-to-one relationships between light and an audio mixer. You can look at a lot of artists and thinkers — going back to Pythagoras, Plato, Aristotle, Cardano, and Newton — who made similar charts, conceiving of various schemes to show relationships between color and sound. It’s a huge subject in science and history. For me, it was a little more simple — I was just trying to grasp the working parts of an experience I’d been having my whole life. In designing Sonified, I could always ask myself, “Is that what this color sounds like to me? Does that audio-visual behavior make sense? Is it the truth?” That kept me on course.

When I felt better,  I rolled up my sleeves and recorded 500 short pieces of music. I selected which ones were the most visual, and which worked together to create a true sense of what synesthesia is like. That process proved really interesting. What does red sound like? Blue? Green? Pitch black? Pure white light? I felt like I could write music for each color for the rest of my life.

Still from Perry Hall's Material Study, paint and ferrofluid, 2006

Still from Perry Hall's video series Material Study, paint and ferrofluid, 2006

In general, the darker something is, the lower it sounds; the brighter, the more high-pitched. Blue and violet are the lowest colors to me; orange and yellow are the highest. Each visual element also has a temperature — orange, reds and yellows are hotter, while greens and blues are cooler.  Each visual quality or color is a timbre, a sonic texture, a pulse, a wave, an ambience, building off of the harmonic series, like stacking harmonics off of a string in a variety of timbres over many registers. But the most important thing is that the visuals and sounds are synched, locked, translated — in a one-to-one relationship in their behavior, proportion and dynamics.

I came to think about synaesthesia more poetically than scientifically. Instead of trying to create an app that was like a Geiger counter, I realized it was better to try and create something more like a guitar — a musical instrument rather than a scientific one. When something felt very subjectively right and truthful to my own experience of synaesthesia, I went with that, regardless of whether it fit into a pre-existing template of how “light relates to sound.”

Silberman: What kinds of feedback have you gotten from Sonified users?

Hall: Just about every person smiles when they put the headphones on and start moving the camera around.  How could they not get it? It’s what they do all the time anyway, seeing and hearing. They’re just doing it in a different way. I get the sense that they’re feeling something I’ve felt. That’s the greatest thing I can hope for, as an artist and a person — that we are all connected, sharing something we were a part of all along, but never knew about until that moment.

Nabokov's typewriter

Nabokov's synaesthetic engine

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Art, Books, Culture, Drugs, Interview, Multimedia, Music, Neuroscience, Personal computing, Photography, Science, Technology and tagged , , , , , , , , . Bookmark the permalink.

Comments are closed.