James Ellis post of Fri, 29 Nov 2002 16:46:34 -0500 to the iota-centre email-list.

Color/Sound one to one, 10 fingers, ten toes, and a bee

I hope I'm not being too redundant with what others have posted here about "one to one" correlation of sound to image, but I (and many I know) have given this topic much thought and discussed it endlessly. There are a few conclusions that we have come up with, both in practice and theory on the creation of real-time music related visuals. Some I don't agree with, some the panel is still open, and on others I take them for as much temporary truth as I can assign to them. I can only truly discuss my own temporary beliefs. Perhaps someone might find these insightful, or add new dimension to these... both I hope. Or who knows, perhaps even kindly discredit them.

Odorama vs. Synesthesia :
There was a movie created in the late 1950's called "Scent of a Mystery". This film (I've never seen or smelt it personally) had scent pumped through pipes into the theater at certain key moments of the movie. This process was known as "Smell-O-Vision". I'm not quite sure why it didn't work out. I imagine it was quite expensive, and that there was no real means to negate previous smells enough to cleanse the olfactory pallet.

John Waters released a film later called "Polyester" (in Odorama) that came with scratch and sniff cards that had the smell of shit, sperm, bleach, etc.. I believe that the audience would take scratch/sniff cues from the screen at specific moments in the film. A cute trick to add gross scent to heighten the overall shock value of the film.

I certainly think this area of sensory immersion has tremendous future possibilities (especially with new scent synthesis technologies), however I'm going to primarily use them as an example of why "one to one" correspondence of visual and sound is a limited approach to a larger overall experience. I believe that the "See and hear at the exact same time" falls into the same category as "see and smell at exactly the same time", and is about as effective (in the case of the color piano) as John Waters scratch and sniff movie. It is a great starting point, but it must be more fluid I say.

The idea of hitting a keyboard frequency and creating a specific frequency of light (or Chakra frequency) simultaneously is certainly a worthwhile endeavor to say the least. However I believe that aside from the goal of purely seeking to induce trance by the rigid synchronization of these two elements through minimizing their scope and interrelation, there might not be as much fertile ground as some might think here. I admit that I could be wrong on this, and if anyone can truly show me a working example of this (without me spending my life savings flying to shows all over the world) I'll gladly "eat my hat". I believe it's an interesting experiment in scientific/mystical cross referencing, but I fear it might be nothing more than an "Om chant"... not to dismiss the power of an "Om chant", I just believe that there are many different ways of experiencing/communicating with existence, and I don't listen to "Om chants" on my stereo or wish to do them all day long. I am certainly for the inducing of trance, and such a formula may very well do that, as well as helping to establish an initial "starting point/vocabulary" for the artist/audience, but where does it go from there. Some might say it's infinite, I would disagree, I believe it's too limiting.

C/Yellow E/Blue G/Pink
Limits of course can be a great frame work in which extraordinary creative pieces can grow, but "bong color" simultaneously I believe has very little to do with how we take in sensory information in general... not to mention that I believe almost any similar system not based on hertz would have a similar effect.

I'd like to share a seemingly silly fictitious example perception, and why I believe "one to one" is not the be all... this leads into tangents which I can't ignore, but will hopefully yield some good fruit.

When I say... walk outside, perhaps I see what's around me as a whole, but my mind has not really found anything to truly focus on yet, so I'm kind of lost in my thoughts about how I should be cleaning or house. My eyes reflex mechanism has me squinting because the sun is bright, so my thoughts almost immediately flash that "gee it's bright out here"... and then I forget about it. Suddenly there is a sound behind me, I turn to where the sound is emanating from... but it's not there. I know this sound... "airplane", I then visually scan the sky and see an airplane. I am then able to connect the sound to the plane even though they are apparently emanating from different locations, due to the delay inherent in traveling sound. I'm now staring up at this plane, bored because I've figured out what it was, when a bee passes in at some distance from of my face. Through life's training, I immediately get defensive, but then realize that the bee is no threat due to it's distance, so I relax. At first I can't hear the bee, but then it approaches closer, zips out of my scope of vision and past my ear (I can hear it and I get a bit defensive momentarily) and I then turn my body to visually attempt to find it's flight path. I see a bee that looks like the same one over near a beautiful flower. I determine that this is indeed the same bee, but now my primary focus is the flower with the bee now secondary. I decide to buck my minor defense mechanism, and approach the flower, the bee leaves and I touch the flower... so nice and soft. I then sniff the sucker, and it smells absolutely horrid to me, but I know my girlfriend, and I bet she would like it's smell... but she also may be allergic to it's pollen. ______________________________________________________________

So, what we have here is a very simple scenario which encompass:

predetermined mind state:
"Should be cleaning my house". Lost in thoughts, almost oblivious to what I have seen so many times before in my yard. In fact, all sensory stimuli is basically unprocessed, and only considered on a immediate physical need... such as walking, personal safety. In some of my performances/screenings I feel that the audience may be dwelling too extensively on preconceptions of "what has been experienced before" or "how they should be cleaning their house". Part of my job is too help them out of this trapped perspective and into my world... or attempt to fathom their mindstate, and give them an airplane or bee to focus on... before I push the envelope.

controlled physical activity that has now become almost intuitive:
"Me walking". Trained coordination and rhythm are not thought of per se, but they are under my control, and allow me to explore my surroundings. These inherent rhythms and physical activities can not be understated in aiding the artist/audience in their involvement and mindstate when encountering art. It is part of the physical automated song that leads us to new experience. Travel, Dance, Driving, Reading, Breathing, and many others... all have physical rhythms in which we can use to "ground" (or otherwise) ourselves or the audience in our works.

inborn reflex eliciting a passive thought:
"eyes squinting because the sun is out.... gee it's bright out here". Proof that sensory stimuli is capable of evoking a response on a physical level, which in turn has a direct parallel in immediate thought. Manipulating thought attached to reflex is of personal interest to me, but thought doesn't necessarily have to be so direct. Could a strobe light then be called "Reflex art" ?

shifts in focus (and nonfocus) of visual stimuli: "The airplane". A pervasive familiar sound that grabs attention and beckons to be attached to, (and mentally confirmed by) visual attention. Here is a case where "one to one" correspondence does not apply in real life (although art doesn't really have to have real life as it's reference point). The sound was heard first, and then seen.

apparent discrepancies in sound/visual placement and forced integration of sound/visual stimuli through past experiences and judgment of distance and speed:
Even though the airplane was heard first, and then this lead to the bodies movement to visual find it, once it has been found, the sound is delayed from the apparent location of the visual. The mind is hard wired to accommodate for minor delays in sound, this aids in the immediate judgment of distance. However when such great distance is accompanied by such speed, it then takes a bit of mental effort to attach the two. Foley artists know very well the principles of this interplay and use it to the overall benefit of simulating visual-speed/sound-delay, and even sweeten it with such things as Dopler. This to me as a visualist, seems like a nice starting point to be expanded upon, and then twisted around to defy all it's inherent rules. Again, no "one to one" correspondence.

boredom incurred by the solving the of the visual affirmation of the creator of the sound:
"Look it was the airplane that made that sound, I'm bored, move on." A certain amount of mystery, will always help keep one intrigued. "What's making that sound?" In the strictest sense, "one to one" correspondence does not have this in its approach to audio visual interplay. You always know that the sound and the visual happen at the same time.

The almost complete mental focus on an attractive pervasive visual stimulus that is devoid of sound:
"Again the sonic mystery has been ended by the airplane, so the mind turns off its visual thought focus, until something jumps at it and demands its attention... the bee". The relatively fast movement (compared to its surroundings) and its immediate proximity are probably the initial key factors in the minds engagement on this visual stimulus. This is then reinforced by the issue of personal safety.

Once again, depth is a key factor in creating mental focus. The relationship of the angle of the two eyes creates numerous convergence points (to choose from) in which they tend to focus best at the point of interest. Once a point of interest has been established, most everything else outside of that area, tends to blur a bit as it falls away. I have seen this effect so nicely used in films. Can the same be said for ears? Waves enter the ears, but how do we mentally interpret them, and what happens to our focus of the larger soundscape when we focus on a particular area of space? An experiment is in the making here for me, perhaps there is a one to one of my own... but back on track. As a primarily non-representational artist, I certainly wish to make more use of this idea of convergence point... and of course, turn it on it's ear, and ignore it whenever I choose. Also this is a case of "no sound, but visual".

Trained defensive mechanism overridden by almost instantaneous evaluation of personal safety:
"A bee... will it sting me... nahhhh, it's far away... your safe, keep and eye out." There is such an overwhelming amount of sensory information that is inborn or that we have been trained to interpret and evaluate in a specific way, and yet it often conflicts. Certain vague conclusions can be drawn on how to manipulate this for artistic means. You can say that a "bee" when it is of threat, could be Orange, but when there is no immediate threat, it's Blue. It depends on the circumstance. The same can be said of note value, timbre, etc.. Circumstance dictates the emotional color. Perhaps there is an overlap, but when does the western scale of pure visual color have the same dynamic range as that of music? I haven't seen it. The depth of variation through combination of notes of music, in my humble opinion, it far far far greater than blank sheets of color performed in succession. Perhaps it's just me, but I don't see it. This thought process of the bee (again in my opinion) could be presented much more affective through sound by itself, where as a pure color sheets by themselves could be affective... but not nearly so. Together, would the two really add anything to each other? I doubt it. Perhaps some of you can change my mind by giving me examples (performed). I really must say, that the notion of the color-piano/organ is an interesting one, but it seems to cut its visual options too small. Too much adherence to a formula... and not nearly exciting enough.

the visual stimulus being replaced by auditory stimulus from the same source:
"The bee was seen, then disappeared from site, but was heard". Another reason why "one to one" is not enough. Shifting of sensory input for the same stimuli.

conclusions drawn based on recent sensory information:
"Can't hear the bee, can's see the bee, it was headed in that direction... ahhh that must be the bee". Here the observer gets to come to some conclusion (based on various sensory information obtained recently) to help fill in sensory gaps with thought and reference. In the hollywood movies, the audience sometimes is allowed to come to some conclusions before the actor is. Neat trick. Again a bit of a mental game, that once made aware of our constant use of it, can be manipulated creatively. Magicians love to use this for misdirection, why should we not use it as illusionists and sonic manipulators? In "one to one" there is not nearly as much room for misdirection and conclusion.

refocusing of point of interest:
"The bee is cool, but the flower is cooler dude". Enough said.

visual gratification in conflict with scent gratification:
"Da flower sure is purdy, but it smells terra-bull!" A pretty sight is not always a pretty smell. A pretty visual, is not always a pretty sound. Check out the Oscilloscope. You could say a "color combinations of the light piano adequately convey such attributes as dissonance"... perhaps. Again seems too limiting in it's emotional scope, even if it adequately can contribute on some form of "frequency relation" to the sound.

evaluation of experience of another's sensory gratification as well as inborn physical aversion:
"This flower smells terra-bull ta me... but I bet my gal Bessie Sue woulda love it's smell... if'n it don't make her sneeze." We all have different taste in music, art, food... and even different physical reactions. It's safe to say that certain things are poisonous to just about all of us, and that candy is sweet. How about aversion to hearing screams of agony... is that trained or inborn? What about that twisted fuck who enjoys screams of agony because he likes to hurt other life? Or what if a someone not mentally disturbed just found screams of agony to beautiful?
Now that I've written all that... there is more...
yes it's true.
You see, I think to sum this part up:
"One to One Correspondence" (in general) does not allow for enough variation between triggered sources. It can work (and does) just fine. I think I personally prefer taking it to the next step in composition... counter rhythms. I also personally need more than just colored lights... or I'll just go see a great light show... but even that is not strictly "one to one" and is all the better for it. Primarily performed separately. [ AuzGnosis's emphasis ]

On the subject of Synesthesia:
it seems to occur when the mind is attempting to interpret something that is beyond it's normal scope of separate sensory interpretation... so the senses blend... to help give a deeper perspective, much like dream or hallucination. To see color and hear sound, pretty cool, but I don't think it strictly adheres to any given formula. My personal research says "No". To create such a formula based on Hertz and apply it to a piano as sheets of color, is a cool creative exercise, but that's all it comes across to me as... no golden answer there... perhaps one small clue. To do so is in my opinion to ultimately create the scratch and sniff card from the film "Polyester"... just more abstracted.

This idea of going beyond "one to one" also leads into the notion of "Ten fingers". The argument is that there are not enough fingers to manipulate real-time animation parameters. This can be true in some cases. Obvious solution number one... "foot triggers". Ok ok, that will only get you so far. The solution we found at Derivative for Touch use in the Rush show, was various short sequence parts that can be triggered, and then different elements can be overridden live when you choose to. When you are done playing that element live, you return to normal sequence. You can also have entire visualist "Band". Numerous persons contribute various elements of the visual composition which can then be composited together by the mixer/conductor. Say one person can play "visual atmospherics" for the first part of the song, and then progressively change into something more rhythmic, and then something that is almost dancing and demanding attention. In this manner you could compose songs with musicians that left plenty of sonic and visual space for each other to fill, devoid of cluttering each other. The visuals might play the 2 and 4 beats, and the musicians play the 1 and 3. Then you trade. Then the visualist go off into chaotic rhythms, while the musicians are orderly. The small light flashes, and then a delayed beat. The slightly larger light flashes again, less delayed beat... it's getting closer.

wellll I'll stop now
As long winded as ever
Blah Blah Blah
I could be wrong

James Ellis vimeo.com/album/1877924


[about] [Legals] [home] [Sitemap]

This Site is constantly evolving, so please forward comments or questions regarding this site to webmaster:
© Copyright 2003 W. Shawn Gray, All rights reserved
 Use of this web site and content from it is subject to our Legal Notice and

Site created AuzGnosis P/L