Inside the Mind of a Synaesthete

"The Figure 5 in Gold," by Charles Demuth, 1928

The Figure 5 in Gold, Charles Demuth, 1928

“I have this rather freakish gift of seeing letters in color,” novelist Vladimir Nabokov told a BBC interviewer in 1962. “It’s called color hearing. Perhaps one in a thousand has that.”

The Russian-born author of Lolita, Pale Fire, and other exuberantly witty books claimed that when he was a child, he saw the number 5 as red, and that he had continued to perceive numbers and letters as having their own distinctive hues.  The interviewer asked Nabokov how the initials of his own name appeared to him. He replied:

V is a kind of pale, transparent pink: I think it’s called, technically, quartz pink: this is one of the closest colors that I can connect with the V. And the N, on the other hand, is  a greyish-yellowish oatmeal color. But a funny thing happens: my wife has this gift of seeing letters in color, too, but her colors are completely different.  There are, perhaps, two or three letters where we coincide, but otherwise the colors are quite different.

It turned out, we discovered one day, that my son, who was a little boy at the time — I think he was 10 or 11 — sees letters in colors, too.  Quite naturally he would say, “Oh, this isn’t that color,  this is this color,” and so on. Then we asked him to list his colors and we discovered that in one case, one letter which he sees as purple, or perhaps mauve, is pink to me and blue to my wife. This is the letter M. So the combination of pink and blue makes lilac in his case. Which is as if genes were painting in aquarelle.

Nabokov’s “color hearing” — a curious neurological phenomenon known as grapheme-color synesthesia — also found its way into his books. In Bend Sinister, the protagonist, Adam Krug, says that the word “loyalty” reminds him of “a golden fork lying in the sun on a smooth spread of pale yellow silk.”

In his autobiography, Speak, Memory, Nabokov launches into a virtuoso synaesthetic reverie: “The long a of the English alphabet… has for me the tint of weathered wood, but a French a evokes polished ebony. This black group also includes hard g (vulcanized rubber) and r (a sooty rag being ripped). Oatmeal n, noodle-limp l, and the ivory-backed hand-mirror of o take care of the white… Passing on to the blue group, there is steely x, thundercloud z and huckleberry h. Since a subtle interaction exists between sound and shape, I see q as browner than k, while s is not the light blue of c, but a curious mixture of azure and mother-of-pearl.”

One of the ravishing pleasures of reading Nabokov is sensing a deep rightness in his word choices (even in English, which was his second language) that goes beyond having a knack for finding le mot juste to make his prose cohere at every level: phonetic, orthographic, and semiotic. Surely the atypical wiring of his brain gave Nabokov an advantage in his quest for this comprehensive unity.

Synaesthetic alphabet

Synaesthetic alphabet

Few writers have mapped this uncanny phenomenon with such obsessive precision, but the gift of multiplex senses turns out to be not as freakish and rare as Nabokov believed. Researchers have learned that even chimpanzees associate low notes with darker colors with high notes with brighter ones. In a recent paper in Psychological Science, David Eagelman of the Baylor College of Medicine argues [PDF link] that synaesthetic ability falls along a spectrum, the end product of multiple processes of neuronal excitation, inhibition, and pruning in the brain gone awry, “all of which happen to converge on the similar result of unusual perceptual or cognitive pairings.”

Another recent study raises the possibility that we are all born with a capacity for sensory crosstalk that diminishes as we get older and our neural networks are streamlined for greater efficiency. Nabokov also believed that we are all born synaesthetes, but because science hadn’t yet elaborated the concept of neuroplasticity, he blamed the loss of the gift on “stupid parents” telling their kids, “It’s all nonsense. An A isn’t black, a B isn’t brown. Don’t be absurd.”

In fact, however, hacking the firewalls between senses may turn out to be a useful skill that parents could teach kids who were not born that way. In an ingeniously designed blog post, Macquarie University autism researcher Jon Brock discussed a recent report by V.S. Ramachandran in Neurocase [PDF link] of a young man with Asperger’s syndrome who was instructed to associate emotions with colors to improve his social perspicacity. As he got older, the young man learned to gauge how he felt about a person by the color of the “halo” around the face.

Nabokov was not alone in his conscious employment of his gift to serve his art. Master painter and digital artist David Hockney relies on his synaesthesic abilities to generate hyper-vivid images that glow with an almost child-like visual innocence. He told author David Burton that when he was designing a set featuring the image of a tree for a production at the Metropolitan Opera of a piece by Maurice Ravel, he listened to the relevant section of the score and “the tree painted itself.”

Felled Trees by David Hockney

Felled Trees by David Hockney

Likewise, the genre-stretching jazz, folk-rock, and avant-garde music of Duke Ellington, Syd Barrett, Alexander Scriabin, and Oliver Messaien [PDF link] was allegedly energized by each composers’ bimodal perceptions. In his epic multi-volume manifesto Traité de rythme, de couleur, et d’ornithologie (“Treatise on rhythm, color and bird song”), Messiaen described chords as “blue-violet rocks, speckled with little grey cubes, cobalt blue, deep Prussian blue, highlighted by a bit of violet-purple, gold, red, ruby, and stars of mauve, black and white.”

For a drearily mono-sensory person like me, it’s tough to read these accounts without feeling a (sour-apple green?) twinge of envy. What would it be like to live in a world in which each prime number generated its own harmonics, every letter of the alphabet was associated with a characteristic odor (the freshly laundered scent of L, the fulsome perfume of Q), and Miles Davis’ “Flamenco Sketches” (on Kind of Blue, natch) shimmered like an iridescent watercolor over the heads of the bored baristas at the local coffee shop?

Color music notation

Color music notation

Now a prolific multimedia artist and writer named Perry Hall [Flash required], who was born with his own version of Nabokov’s quirky gift, has developed an iPhone/iPad app called Sonified that enables even those low on the synaesthetic spectrum to experience light, colors, and movement morphing into sounds.

I first became interested in Hall’s work seeing a series of haunting HD videos made in 2006 that he called Material Study, featuring light dancing on the surfaces of ferrofluids that surge and swell like some kind of protean lava. While convalescing from a bout of Lyme disease, Hall decided that he needed to set his synaesthesia loose in the wild, as he puts it. He and his digital collaborators developed software that siphons the luminance and color values from the video cameras in iPhones and iPads (only later-generation devices like the iPhone 4, 4S and iPad 2 will work correctly) and uses them to trigger stereo samples from a library of CD-quality audio composed for the purpose.

Photo of Perry Hall

Perry Hall, artist, synaesthete, and designer of Sonified

When Hall — who helped create the lush “painted world” sequence in What Dreams May Come, the 1998 film starring Robin Williams — told me about Sonified in email, I knew I had to try it myself. After downloading it from the App Store, I boarded a streetcar here in San Francisco, slipped on a pair of headphones, and aimed my phone out the window just as the train streaked past a row of brightly painted Victorian houses, accelerating through shafts of sunlight and shade on its way into a tunnel.

The effect of the audio-visual-kinesthetic link-up was unexpectedly profound. Instead of feeling like Sonified was imposing its digital soundtrack on the world, I felt I was accessing a layer of reality that is normally hidden from us. It was like a little dose of Morpheus’ red pill in The Matrix.

Sonified screen shot

Sonified screen shot

To give potential users a foretaste of the experience, Hall has uploaded videos to YouTube here, here, here, and here. But experiencing Sonified second-hand rather misses the point. The thrill of using the app is having it respond to optical nuances in real time as you move through spaces that come alive in new and surprising ways. Hall’s ethereal sonic palette may be a bit New Agey for some tastes, but the software offers a teasing glimpse of how much more we could be doing with these powerful multimedia platforms in our pockets. Sonified takes the often mundanely-applied concept of augmented reality (we were promised Terminator vision and got Plaxo QR codes instead) a step closer to the radical departure from sensory business-as-usual that 18th century multimedia pioneer William Blake described in The Marriage of Heaven and Hell:

How do you know but ev’ry Bird that cuts the airy way,
Is an immense world of delight, clos’d by your senses five?

Glad Day or the Dance of Albion, William Blake, circa 1794

Glad Day or the Dance of Albion, William Blake, circa 1794

I asked Hall, who lives in western Massachusetts, what it’s like to walk around every day with naturally augmented senses.

Steve Silberman: When did you figure out that you experience the world differently from most people?

Perry Hall: I’ve been experiencing two different kinds of synaesthesia since I was about 10 years old. The first is “colored hearing,” where sounds become very visual; and the other is a milder type of synaesthesia that is more directly connected to Sonified, in which images, color, and spaces like landscapes inspire tones and sounds. Both types are very emotional for me, and define a profound beauty that compels me to write music or create visual works of art that are reflections of it.

When walking through certain kinds of terrain — usually those involving vast spaces or interactions of light and color — these tones will become present. It’s usually in a natural environment, such as a beach, landscape, or other place where deep space and light can play, stretch, and unfold with a kind of purity. But it’s always there, at least in the background.  I was just in Wales taking a train along the Irish Sea, the landscapes that Turner captured in his paintings. I was filled with a lot of sounds as I moved through this environment.

Wreckers Coast of Northumberland, Joseph Mallord William Turner, circa 1836

Wreckers Coast of Northumberland, Joseph Mallord William Turner, circa 1836

I feel an intense identification with these spaces, as if the distance and distinction between myself and the environment is collapsing, and I identify with the sky or landscape and lose myself. The sounds that come from this environment blur the distinction between seeing and hearing into a single point. Visual dynamics create sonic dynamics — the two are joined, like a mirror, with visual qualities on one side and sonic qualities on the other. When the light slowly fades, the sounds fade; as the light shifts, increases, or decreases, the sounds shift equally. A circuit is made between seeing and hearing, but it’s not seeing and hearing, it’s see-hearing, seering, hearseeing — something distinct from merely a combination of the two senses that we think of as separate.

Silberman: What inspired you to try to make these experiences available to others?

Hall: Eighteen months ago, I was diagnosed with Stage 2 Lyme disease. I had it for over a year, including a fever that lasted for four months. Finding myself in bed, wondering what was going to happen, I had time to think. I realized it was important for me to make this strange and wonderful experience accessible to other people. I came up with a means of creating a “synaesthetic engine” that could translate light and color into sound. I wanted to put this engine into a camera anyone could use anywhere, rather than putting it in a gallery or museum. I wanted people to be able to walk through a desert or a city, to lay in bed or sit on a train, and have the same kind of experiences I can have anywhere. Knowing I was stuck in bed, a friend gave me an iPhone to play around with, and I saw it had all the pieces that I needed to set synaesthesia loose in the wild.

Silberman: How did you go about designing the software?

Hall: First I created the design documents — a set of charts showing one-to-one relationships between light and an audio mixer. You can look at a lot of artists and thinkers — going back to Pythagoras, Plato, Aristotle, Cardano, and Newton — who made similar charts, conceiving of various schemes to show relationships between color and sound. It’s a huge subject in science and history. For me, it was a little more simple — I was just trying to grasp the working parts of an experience I’d been having my whole life. In designing Sonified, I could always ask myself, “Is that what this color sounds like to me? Does that audio-visual behavior make sense? Is it the truth?” That kept me on course.

When I felt better,  I rolled up my sleeves and recorded 500 short pieces of music. I selected which ones were the most visual, and which worked together to create a true sense of what synesthesia is like. That process proved really interesting. What does red sound like? Blue? Green? Pitch black? Pure white light? I felt like I could write music for each color for the rest of my life.

Still from Perry Hall's Material Study, paint and ferrofluid, 2006

Still from Perry Hall's video series Material Study, paint and ferrofluid, 2006

In general, the darker something is, the lower it sounds; the brighter, the more high-pitched. Blue and violet are the lowest colors to me; orange and yellow are the highest. Each visual element also has a temperature — orange, reds and yellows are hotter, while greens and blues are cooler.  Each visual quality or color is a timbre, a sonic texture, a pulse, a wave, an ambience, building off of the harmonic series, like stacking harmonics off of a string in a variety of timbres over many registers. But the most important thing is that the visuals and sounds are synched, locked, translated — in a one-to-one relationship in their behavior, proportion and dynamics.

I came to think about synaesthesia more poetically than scientifically. Instead of trying to create an app that was like a Geiger counter, I realized it was better to try and create something more like a guitar — a musical instrument rather than a scientific one. When something felt very subjectively right and truthful to my own experience of synaesthesia, I went with that, regardless of whether it fit into a pre-existing template of how “light relates to sound.”

Silberman: What kinds of feedback have you gotten from Sonified users?

Hall: Just about every person smiles when they put the headphones on and start moving the camera around.  How could they not get it? It’s what they do all the time anyway, seeing and hearing. They’re just doing it in a different way. I get the sense that they’re feeling something I’ve felt. That’s the greatest thing I can hope for, as an artist and a person — that we are all connected, sharing something we were a part of all along, but never knew about until that moment.

Nabokov's typewriter

Nabokov's synaesthetic engine

Related Posts Plugin for WordPress, Blogger...
This entry was posted in Art, Books, Culture, Drugs, Genetics, Interview, Multimedia, Music, Neuroscience, Personal computing, Photography, Science, Technology and tagged , , , , , , , , . Bookmark the permalink.

44 Responses to Inside the Mind of a Synaesthete

  1. Pingback: ¿Sinestesia? ¡Hay una aplicación para eso! | Noticias CEU

  2. Pingback: » ¿Sinestesia? ¡Hay una aplicación para eso! (Pablo Gallego Falcón)

  3. Pingback: Somewhere Over the Rainbow, There Words Fly | Brigolante Guest Apartments

  4. Pingback: Sound's Impact on Taste Can Help us be Better and Humble Winemakers

  5. Pingback: Inside the Mind of a Synaesthete | Emily Tries to Make Things

  6. Linda says:

    This is very much so an experience that I have as well with music. As a child we were encouraged in some of our art classes to “paint the music” and as an early musician (keyboards–organ, piano, etc.) I could paint the most beautiful notes in my mind’s eye. Sometimes they would come out in the colors, sometimes, I just couldn’t find the right shade/hue and it would cause frustration that the teacher understood (for the most part, she’d try to help, but her view was so different from mine).

    Also, this ability causes much confusion when I read a book. I have all the characters with their descriptions and attributes all in my head – then Hollywood comes along…. Messes the whole idea up and I can’t stand the movie. Don’t know if this is anything at all like what we’re talking about, but it sure does make or break a film for me if I’ve read it first!

  7. Pingback: Well, Colour Me Tuesday. | Lynley Stace

  8. Pingback: Link love: language (40) « Sentence first

  9. Pingback: Colorful language » Gydle

  10. I-) says: Painting With Sound – photos by Martin Klimas

    certainly very close to music experienced as color, especially the ones of electric miles davis…


  11. Pingback: Synthetic Synesthesia « benjaminaaron

  12. Jean Lamb says:

    I have an entirely different brand of synesthesia–I have sound/tactile, as if the sound is coming out of me (I can also ‘feel’ exactly which part of the throat/nose/whatever is being used when someone sings). This makes choreography pretty easy for me, of course. Although I do have some other effect with sound–Madonna’s vocals always feel different shades of blue, and Anita Baker’s voice tastes like honey and cinnamon on slightly burnt toast.

    Listening to “Rhapsody in Blue” by Gershwin is like riding a roller coaster–whee!

    • Perry Hall says:

      Jean, this is really interesting and again, so many different forms of synaesthesia– it always draws me in to hear another person’s account. What a voice “looks” like is a unique kind of thing for me; a voice, when I hear someone speaking, is usually like a line drawing, with a sense of line; different widths, shapes of line, mostly in blacks or grays, like pencil; but occasionally someone with a unique voice: the “line” of their voice will be more 3-dimensional, or will be made out of some metallic color; sometimes colors will be a part of the sound of the line. Then, if the voice begins singing, its shape becomes more dynamic, growing and shifting, fanning out, spreading, turning into something more full; tissues, volumes, ribbons, or even light, depending on the voice. Lower voices become darker, thicker, like a material made of mud, tar, black smoke, even dark stone, with glints or sprinkles of color within, if the person’s voice has a rich timbre.

      • Zube says:

        Perry Hall? Of the adjunct Bennington prof; Perry Hall?
        The teacher who helped form the best video project I made during my four years at Bennington?

    • My synesthesia is also sound/tactile. For me it is as if sounds are carresses/touches/scrapes on various locations. I love music almost as much for the sound as the caress.

      I took the “Embracing the Wide Sky by Daniel Tammet” to realize the sound/touch fusion was unusual. I can’t remenber a time without it.

  13. Pingback: Synesthesia: experiencing one sense as another, different sense | Chainsaw on a Tire Swing

  14. CB Sorge says:

    I wish people knew about and subsequently shared their synaethesia more, as it’s fascinating how many different combinations there are, and how that effects lives. I have the common grapheme-color association (as well as day/week/month) and am a very visual person, an illustrator. I also have always associated personalities or temperments to them, especially numbers (5 is an orange doofus that I can’t stand).

    But one of my earliest memories is tasting cartoons. I lost that ability, sadly, as my brain pruned away the unnecessary connections.

  15. Patty says:

    Very interesting article! It did occur to me when I saw the picture of the colored letters, that maybe some of this is due to associative memories of childhood books? My older daughters are in their mid-forties now, but to me, broccoli is always “Bobo Broccoli” from the game they had as small children!
    Probably any child has these associations, but they get lost in the clamors of growing-up, which is sad. We should always embrace and enjoy our differences; what a dull old world this would be if we all felt, heard, saw, and thought the same things :-)

  16. Reynold says:

    Reading this brought up childhood memories. The agonizing boredom of long road trips would be dramatically relieved when the droning sounds of the car would change into and endless stream of symphonic sound. I wanted to ask my parents, “Can you hear the music?” but I knew better than to do that. Vivid colors accompanied the sound against a black background.

    Does this kind of experience have value beyond a private, intense aesthetic pleasure? I believe it does. Our personal boundaries dissolve, or extend out into the universe. Lust, ambitions, desire for power and wealth also fade into non existence. Our survival may depend on it.

  17. Perry Hall says:

    Jenn- ofcourse, there’s no one way to perceive colors and light as I said in the article, and I think it’s fantastic that there are all these differences in people. The idea with Sonifiefd would be for you, or anyone, to create their own set of sounds and put them into Sonified so that a vocabulary of synesthesias is created, and in the process more films, music and experiences come into the world. Future updates of the app will let you do this, but anyone can contact me and I’ll show them how to hack the software so they can put in their own version of what color and light sound like!

  18. Pingback: Synesthesia, visual music and Sonified « Dr. Electro's Mixtapes

  19. Andrew Wright says:

    I always thought that what I perceived in my mind’s eye when dealing with numbers, sounds, days of the week, months, etc… was normal until I saw a Night-Line segment years ago about it. I couldn’t believe it – I thought it was a normal thing that everyone “visualized” their senses as I did rather than it being an “augmented” perception. It makes sense to me now of how my life has evolved: I have an Art History degree with a minor in Music, a multi-instrumentalist in a working rock band and a professional sound editor/mixer for film and television. Knowing what synaesthesia is now, makes my life and career make total sense to me whereas before knowing what it was, I couldn’t give any answer as to why I do what I do. Too bad my friends, family and bandmates scoff at me when I try to explain it all to them.

    And YES the opening of Fantasia is a perfect example of how I “see” the world of sound. That is always my go-to example.

  20. Pingback: The Mind Of A Synaesthete: Seeing Letters In Colors | Pennsylvania Seek and Find

  21. Gydle says:

    I always just assumed that everyone else always saw the days of the week, the months, the numbers and letters in color, until someone laughed when I said that Wednesdays were always blue and I realized they had no idea what I was talking about. I’ve been meaning to post about this on my blog, ever since reading David Eagleman’s Incognito. I’m fascinated by the other forms of synaesthesia that I don’t enjoy. It’s weird the way we just assume everyone else experiences the world exactly the way we do.

    I also “see” a word, typed out on the screen of my mind’s eye, when I hear it or even think it. Which is why I have always been a very good speller. I wonder if everyone else does that, too, or is this another form of crossed connections?

    • Andrew Wright says:



    • A. Marina Fournier says:

      Another person who sees words–I used that to good effect when I was having to spell words aloud.

      I also can’t remember most people’s names unless I see them in print (name tags), or I make the effort to see them written in my brain.

      Somewhat offtrack, but an example of my language skills.
      When in college, working at the main university library, I was tasked with getting bibliographic info off books not printed in English. I had little problem figuring out most of them–you looked for the place it was published, and that usually gave you the country and language. When I couldn’t find that, I looked at the language. The cognates, “little words” and the diacritical marks gave me the best hints, as well as some of the cognate words in the text. In my third year there, we had some new student assistants, who said, Oh, I have only had language X class, and I can’t read those. I wasn’t reading them so much as pattern matching, so I thought they were lazy or stupid. *I* had taken only Latinate/Romance languages: French, Spanish, Latin, and Old French, but I was handling any Roman-lettered language.

      BTW, the earliest OF is closer to modern Spanish and to Latin, but I had had a book of OF/MF poetry, so I had had some exposure to OF in relation to MF already. I tried once only to work out a translation in a study group, because they were so slow!

      These days, it’s pretty much limited to matching names with country, to figuring out what language is being spoken by passers-by, thinking in French, and to easily and correctly pronouncing unusual names.

  22. I-) says:

    what ‘Jenn says: February 8, 2012 at 8:50 PM’ is so very true – the sonified music does not match the visuals. synesthesia is very much a personal experience that, at best, you can describe what you are experiencing, but another synesthete experiences it differently.

    the start of walt disney’s ‘fantasia’ when the orchestra turns into moving colors – a good example of synesthesia.


  23. I-) says:

    i ‘see’ music as colors – as mental images, if you will, ever changing. that is why i like certain music and groups, like the grateful dead. listening to the ‘dark star’ on their ‘live/dead’ album is like having a mental orgasm.


  24. Simone Brightstein says:

    What a superb article!

    I’m a lexico-gustatory synesthete combined with some shape synasthesia but, unfortunately, haven’t the gift of colour or sound. Still, I greatly enjoy the enhancements I do have and am enchanted with the idea of Sonified, which I will explore.

    Blake”s “Glad Day”, reminds me of The Fool in a Tarot deck – absolutely magical.

  25. LJ says:

    Synaesthetes aren’t limited to hearing in color. Everything I taste and smell has color, shape, position, and often directional spin. Makes cooking a true multisensory activity.

  26. Jenn says:

    Funny how in watching the sonified videos, I just kept thinking that the music was all wrong for the colors. Like how my sister’s 4 is orange and mine is purple. It’s akin to knowing a secret language that nobody else speaks. You meet others who speak a secret language, but none of you speak the same one.

  27. Pingback: Hang in there guys! « The Book of Ascension Blog

  28. Pingback: Silly Connections Today | Tidbits of My Mind

  29. MariaA says:

    Wow. Congratulations on accomplishing all of that while recovering Lyme, that’s quite an achievement.
    -another synaesthete with Lyme

  30. Pingback: Inside the Mind of a Synaesthete « Color Workshop

  31. Chris says:

    This is an incredible article. I only wish I had an iPhone so I could try out Mr. Hall’s app. Also, now I know why the painted world sequence in What Dreams May Come resonated with me the way it did.

    A friend of mine shared this blog with me because of a post I wrote a few months back: Yep, I’m a Weirdo. Good stuff. I’ll be following!

    • A. Marina Fournier says:

      That was an interesting article–it caused me to comment about my “eyelid movie screen”, and my color associations for each month, which I forgot to mention here.

  32. Sounds like it would be a fun way for non-synaesthesiac folks to see what we are talking about!

  33. Perry Hall says:

    A. Marina, you’re absolutely right, it wasn’t in the exact spot of Turner’s painting, but it was in the neighborhood; it had the same sense of his landscapes which I had looked at so many times, but had never been that close to.

  34. I am in love with this post. Nabokov is my all-time favorite writer and I had no idea that he saw letters in color. That’s so cool.

  35. A. Marina Fournier says:

    At one point, up until maybe 10 years ago, I had strong number to color associations. Letters, no. A lot of music, classical/instrumental, middle-eastern, and a few others, make me see/feel choreography. I know I have watched ballet and other dance forms and disagreed with the steps used, because my brain didn’t see it that way.

    As to stupid parents, we always wanted to explore my very imaginative son’s perceptions, rather than denigrate them, so we asked leading questions.

    I will have to look into Sonified–sounds amazing. I remember quite strongly the painted world sequences in What Dreams May Come. Much of the figural representation reminded me of Michael Parkes’ work.

    You say you were taking a train through Wales, along the Irish Sea, but the Turner you used is on the opposite coast, and farther north, adjacent to Scotalnd, which puzzles me.

    This is a great article, much like your articles on autism: they all address perception and cognition. Thank you!