Historical Context
https://www.theguardian.com/artanddesign/2006/jun/24/art.art
“Poems and paintings became music, and music became poems and paintings.”
While visual music is becoming more popular with the technology available today, there are many historical examples regarding visual music as well. While synesthesia is a natural human phenomena, the implementation for sound and sight linking dates back to the 19th and 20th century (The Guardian). Wassily Kandinsky, a Russian composer and visual artist, captures the beauty of music through paintings and other forms of art. He wasn’t alone in his ideas either. Many other artists implemented similar ideas into their own work, linking sound and music with visuals and paintings of the Romantic era. From these original ideas and inspirations, these artists inspired modern lighting and color for many projects available today. For example, concerts across all genres have many lights that fit the theme of the show and each song. A slow, acoustic song has dim, soft lights, whereas a powerful, high energy song has bright, fast changing lights. In a way, modern concerts are visual music. With the empowering instrumentation and bright lights, concert goers experience music in a way that isn’t tangible at home.
Before the technology for modern lighting was available, Scriabin composed his work “Prometheus: Poem of Fire 1909-10,” which was intended to involve a color keyboard, lighting up the entire concert hall with color. Scriabin had synesthesia, where he would perceive certain keys as different colors as opposed to individual notes being a different color. In Yale’s video on YouTube, they show the implementation of the color organ and how it is performed in concert.
Yet another example of live visual music is “The Firebird” and “The Rite of Spring,” which Igor Stravinsky composed for Sergei Diaghilev’s Ballets Russes company. In these pieces, Stravinsky intended for multiple forms of art to be present during the performance. Painters, dancers, and musicians combine their art forms to perform a giant art masterpiece for an interactive and immersive experience for the audience.
Current Context
Visual Music
In this video, Nahre Sol breaks down how she interprets complex chords. These chords aren’t typically seen in pop music and are usually pretty challenging to analyze and play. To give an idea of normal chords, there are two types to start off with: triads and 7th chords. A triad consists of three notes involving the root, 3rd, and 5th of the key. The root meaning the name of the chord (In a C major chord, C is the root), the 3rd being 3 notes above the root (In a C major chord, E is the 3rd), and the 5th being 5 notes above the root (In a C major chord, G is the 5th). In a 7th chord, we build on the original triad and add a 7th above the root. In a root position C major chord, a pianist will play C, E, G, and B, in that order. Of course, there are all sorts of inversions and qualities of chords to consider when referring to triads and 7th chords.
In this video, Nahre shows more complex chords, involving more notes than a standard triad or 7th chord. She shows this interpretation with color. First she plays the chord with a bowl of mixed colors seen at the top of the screen. Then she reveals the two colors used to compose the chord. In one example, she represents an A minor 6/4 chord with the color blue, and an A major chord with the color red. Finally, she plays the chord again showing the mixed colors and the musical notes laid out on sheet music, with their respective colors. This form of visualization can help beginner musicians understand complex chords and how to build them. By thinking of the chords in colors, rather than notes on a piece of sheet music, people may be able to easily understand music theory.
https://www.wired.com/2011/05/visual-music-paul-prudence/
With the technology that is available today, it is no wonder that visual music has become so popular. In this article, Alice Vincent shows an example of visual music by Paul Prudence and how he creates these visuals. All of the visuals are created with an algorithm based on math and geometry (Wired). The visuals themselves look stunning when paired with the music. He intends for his visual music to be experienced live due to the originality of each piece and visual. As an art piece, this form of music stands as a credible source for visual art made with current technology.
In my original idea for the thesis project, I was extremely intrigued by chromesthesia, the association between sound and sight. In this video, we see a young woman playing violin with bright colors popping up on screen to represent the notes while she gives commentary. The colors blend into the cityscape and environment as if they were a natural phenomena, and not inside one person’s head. After wondering, “what does synesthesia look like,” we get this video where someone with synesthesia can accurately represent what they see through the technology.
Based on this knowledge of synesthesia, I realized that my project would be something different. I didn’t want it to look like this video. The video is beautiful and creates a wonderful understanding for what the condition looks like, but it is not clear what each specific note is. Even after spending years training my ear, I cannot identify the notes or intervals she is playing even when looking at the visuals. After reading some comments on the video, people with similar conditions only see certain colors for particular songs, or a whole orchestra may appear one whole color as opposed to multiple vibrant colors. In my head, I envisioned synesthesia to involve a rainbow of colors ranging in shape, tint, hue, saturation, etc. After seeing this video and reading many stories, I decided to leave the idea of synesthesia as an inspiration, and not the sole focus of my project.
Deafness and Music
“Sound is so powerful that it can either disempower me and my artwork or it can empower me. I chose to be empowered.”
In this enchanting TED talk, Christine Sun Kim discusses the similarities of music and sign language and the importance of social currency in the deaf community. She gives the audience some background and history on American Sign Language (ASL) and its importance today. Her entire life, she was always taught to consider sound as something separate from her. Sound was something that she would never be able to experience and she would be distanced from it. As her life went on, she realized that ASL and sound are not as different as the average person might think. She even acknowledges that she spends more time paying attention to sound etiquette and mirroring hearing people with their sounds. At one point in the talk she says, “In deaf culture, movement is equivalent to sound.” This makes sense because all signs in ASL have movement to them, therefore making “sound with them.” What’s interesting to me is that music also has movement to it, both physically and metaphorically. As a violinist moves the bow up and down, we can gauge how intensely or passionate the piece sounds. Even if we covered our ears, we can still get a general idea of the tone and texture of the piece without ever hearing it. In addition to the physical movement of the performer, music and sound has its own movement to it. The phrasing of each piece contains a special movement that changes based on who is playing the piece. A performer can make a melodic line sound melancholy, whereas another performer can make that same line sound lively and energetic.
She also mentions the similarities between music and ASL, which are surprisingly similar. ASL is visual, so we can typically see drawings or paintings relevant to the language, but we hardly see or hear any similarities between music and visual languages. One great similarity is how music and ASL cannot be fully expressed on paper. (TED talk). Christine goes over the different parameters of sign language, including body movement, facial expressions, speed, hand shape etc. All of these parameters cannot be expressed on a piece of paper like english can. “English is a linear language,” and doesn’t have as many physical parameters to consider. Music is similar to sign language in the sense that it is not a linear language. Sheet music exists, but it does not fully capture the meaning and essence of the tone and quality of the piece. She uses a piano metaphor to explain the similarities, where english is a single note and ASL is a chord constructed of the many parameters to interpret the language. Much like music, if one were to change any part of that chord, the whole meaning would change.
My idea for this project was to drive home the point that deaf people can experience music. They can experience it in a number of ways involving visual and tactile sensations. “You don’t have to be deaf to learn ASL, and you don’t have to be hearing to learn music.” Deaf people do have a voice, they just don’t use their physical one. By recognizing the similarities between sound and ASL, both hearing and deaf people can come together to bridge the gap between the two communities.
Accessible Technology
On How Deaf People Might Use Speech to Control Devices (Jeffrey P. Bigham, Raja Kushalnagar, Ting Hao Kenneth Huang, Juan Pablo Flores, Saiph Savage)
Accessible music
“If I were here playing cello, or playing on a synth, or sharing my music with you, I’d be able to show things about myself that I can’t tell you in words.”
In this inspiring and touching TED talk, Tod Machover showcases his many projects involving music technology and accessibility. He makes the point that music is much more enjoyable when you can create it yourself (TED talk). In my project, I hope to achieve part of this goal by giving deaf people a means to learn the fundamental skills for music without needing sound. We see part of the wonders with Tod’s innovation, Hyperscore, a program that allows users to coordinate lines and colors to create music. By the end of the video, Tod brings in a man named Dan, who has cerebral palsy. Even with his physical disability, everyone can see that through the power of the infrared camera, Hyperscore, and sensors, Dan was able to express himself in ways that words cannot define. This is truly an inspiring piece of work and provides a foundation for what accessible technology should aspire to be.
If technology is accessible, that means that everyone, no matter their disability, can use it. Even today, there are still devices and innovations that do not cater to people with disabilities. In addition to helping this group of people, accessible technology can also make life easier for the average user. Take music for example. The average person can learn music, but it is extremely difficult and some may struggle more than others. But, with a program similar to Hyperscore, people across a wide spectrum of diversity can create something meaningful to them and improve their quality of life.
Ad-Hoc Access to Musical Sound for Deaf Individuals (Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, Suranga Nanayakkara)
An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Musical Display and a Haptic Chair (Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, S. H. Ong)
Fourney, D.W. and Fels, D.I. Creating access to music through visualization. Science and Technology for Humanity (TIC-STH). 2009, 939–944.
Musica Parlata: A Methodology to Teach Music to Blind People (Alfredo Capozzi, Roberto De Prisco, Michele Nasti, Rocco Zaccagnino)
Color Theory
Technical Implementation
Max
After spending one semester using Max, I figured this would be my first approach for technical implementation. I have been able to make arpeggiators, random number generators, and interactive music pieces from coding on Max. For those that are unaware, Max is a program that allows for interactive music and more through patches and cords. After researching on the website, Max does allow for real time graphics, which is exactly what my project calls for. Also, I can potentially do the demo section of the project with Max as well. I am definitely most comfortable and familiar with this program compared to the other solutions.
WebGL
WebGL (Web Graphics Library) was suggested by a classmate who worked with this interface in a previous class. It allows for 2D and 3D graphics in any compatible web browser. While this is an interesting web based solution, I am not confident or skilled in coding, and I fear that my lack of skill will inhibit my progress with my project.
Chrome Music Lab
The Chrome Music Lab is an excellent source for easy music creation. Just from playing around on the site, one can easily make a simple melody in a few minutes. By using color and a visual interface, it gives the user a refreshing experience with music composition. One particular experiment of interest to me is the spectrogram, which is defined as a picture of sound (Chrome Music Lab). It almost looks similar to a heat map for sound, with certain frequencies appearing more red and others appearing more blue. In terms of my project, it’s important to consider these sources that use color to represent music and sound. In regards to my project, I may want it to perform similarly to the spectrogram where the notes occur on a timeline.
Processing
As suggested by Professor Ault, my last option is Processing, a program that allows for visual art coding. Again, this is a great solution for going about this project, but I am not confident in my coding skills.