Vivisi: a System of Musical Illumination

Vivisi: A System of Musical Illumination

Foster Phillips, Department of Industrial Design, Auburn University, USA

Tsailu Liu, Department of Industrial Design, Auburn University, USA

Abstract

Can a musical performance be enjoyed by those with little or no ability to hear? This is the question raised by the research and design of a new generation of musical instruments. It is proposed that an instrument which can incorporate a real-time visual display of the sound it is producing will allow those with different hearing abilities to enjoy a live instrumental performance. The visual display of sound, located within the instrument, will communicate the rhythm, patterns and dynamic ranges of the performance by utilizing the strengths of colored light. The color of the light is directly correlated with the pitch of the musical notes, and the brightness of the light is directly related to volume of the sound. The resulting instrument will allow musical performances to be an engaging activity for all people, including those with a hearing loss that previously would have limited their level of interaction with a musical performance. The instrument also adds to the enjoyment of those with normal hearing ability, by creating a poly-sensorial experience in a place before dominated by auditory stimulation.

Keywords

sound, music, musical, musical instrument, performance, musical performance, violin, hearing, hearing impaired, deaf, hearing loss, synaesthesia, LED, light emitting diode, color, light, amplitude, volume, frequency, pitch

1. Introduction

Musical performances, by their very nature, are geared towards people who are capable of hearing. Could it be possible to design a musical instrument that would allow those who have limited or no capability of hearing to enjoy an instrumental concert?

The violin attracts musicians and listeners the world over. It has been a force in the musical world since its origins in the 1500s. As such, the violin has a long history as a solo and lead instrument, with many of the great musicians using it to display their musical prowess. H.R. Haweis, author of Old Violins and Violin Lore, gives a spectacular explanation of the lure of the violin, “as a tone producing instrument and within its limits it is perfect – every gradation of sound between tone and semitone is attainable, and for no other instrument can this be claimed” [1]. The violin, after five hundred years, is still a compelling instrument, and continues to be used in solo performances and in lead performances within an orchestral setting.

Because the violin has captivated international audiences for years, it was a natural choice to use as the vehicle for bridging the gap between auditory and visual music. By using a strict one-to-one correlation of pitch to colors and by directly relating volume with brightness, it was hypothesized that the rhythm, patterns and dynamics of instrumental music could be shown in a visual format. Another key decision was placing this music illumination system within the body of the instrument so that non-hearing audience members will be fully engaged in the performance. By locating the source of the music illumination system at the source of sound, the connection between the two is strengthened.

An external music visualization would cause non-hearing audience members to look away from the musical performer to see the audio translation, thus missing the physical nuances of the performance and therefore would be disconnected from the rest of the audience and the performance itself. The benefits of the music illumination system located within a musical instrument are a musical performance that engages non-hearing audience members, and creates added depth for hearing audience members by directly connecting the auditory and visual realms.

1. Synaesthesia

Though rare, there are some people who experience the world in a way that most people do not. Synaesthesia can best be described as a melding of the senses. The word comes from the Greek words “syn” which means together and “aisthesis” which means perception. Synaesthesia may be better understood by studying to the word anesthesia, which means “no sensation.” In contrast, synaesthesia means joined sensation [2]. Synaesthetes experience the world in a way that is foreign to most people. Certain sights, sounds, tastes and odors can trigger their other senses causing the world to be quite an interesting place.

One form of synaesthesia, called chromophonia, is especially relevant to this study. Synaesthetes with chromophonia, also known as colored hearing, see colors that are directly related to the pitch of the sounds they hear. The sound-color relationship experienced by synaesthetes with colored-hearing is typically a one-to-one, pitch-to-color association. That is, when a G note is heard, the color blue may be seen. Different synaesthetes rarely experience the same responses to the same stimuli, but multiple tests of a single synaesthete over a period of time are likely to produce similar results.

Jamie Ward, Brett Huckstep and Elias Tsakanikos of the Department of Psychology at the University College London, wrote an article for the February 2006 issue of the medical journal Cortex entitled “Sound-Colour Synaesthesia: To What Extent Does It Use Cross-Modal Mechanisms Common To Us All?” The article’s main premise is that there is a common link between how subjects with sound-color synaesthesia (chromophonia) and subjects without synaesthesia relate the pitch of a note to the lightness of colors [3].

3. Color/Pitch Survey

A survey was developed to define what colors would be used to display the notes played by the violin. A Macromedia Flash based color/pitch correlation test was devised to find if there was a common bond between how people relate colors with notes. A computer was used to administer the test and subjects were encouraged to take the test in a quiet environment using headphones. The test allowed subjects to select one of twelve notes from D4 to C#5 played on a violin. The notes were not in order, nor were they labeled as notes. Extreme care was taken to ensure the notes played were exactly in tune, with no vibration and at a constant volume. Subjects were able to play each note as many times as they preferred and to change their answers at any time. The colors chosen for the test were taken from common colors used in color scales proposed in the past. Subjects were assured that there were no wrong answers. Once subjects were satisfied with their color/pitch combinations, the data was recorded.

Figure 1. Playing a Note, Selecting a Color, and A Completed Test

To play a note using the test (Figure 1), subjects used a mouse to click on any number, 1 through 12. The gray box around the number would be highlighted when the cursor was hovering aboveit, and once the box was selected, the note would be played. Notes could be played at the same time to test musical harmony against color harmony, and could be played in any order or combination.

Colors were selected in a similar manner. When the computer mouse hovered over a colored box, the white border around the box would enlarge. Once the box was selected it would be marked by the number of the last note played (Figure 1). Subjects then would continue the test until all boxes were marked by an individual number, indicating the note that was selected.

The test was administered to 116 subjects. The subjects were adults, aged 18 years and above. No data was collected regarding gender, ethnicity, musical ability or lack thereof.

3.1 Results

The most common answers can be seen in Table 1. D# paired with Maroon was the most common response with 31% of the subjects identifying that pair than any other. Sky blue was the least identifiable color, with nine notes being identified with it somewhere between 7% and 13% of the time.

Table 1. Results of the Flash Based Color/Note Correlation Test

There was not a significant similarity in the answers of different subjects, but one theme began to emerge. After testing, subjects would seek reassurance that they had answered correctly, and begin to justify their responses. A typical response became, “I tried to put bright colors with higher pitched sounds and darker colors with the low tones.”

3.2 Analysis

A one-to-one color/note correlation is a personal connection made by an individual. Out of 116 participants, no two subjects created the same color scale. There are over 400 million different ways the color test could have been answered, so there is no surprise that there were no repeated answers. The only way there would have been repeated answers would have been if there existed a universal relationship hidden in the depths of the human brain. No subject reacted with extreme confidence in his answers. Also, no one reported the problem of the wrong colors being included in the test. The twelve standard colors selected for the test may have frustrated a synaesthete because synaesthetes typically have specific color and pitch pairings. Since this was not an issue, it was concluded that no synaesthetes were surveyed.

Although no universal one-to-one color relationship was established from the test, there were some trends that emerged. Similar results were found as the study conducted by Jamie Ward. Ward found that synaesthetes and non-synaesthetes share a common bond of choosing darker colors for lower tones and lighter colors for higher tones. The non-synaesthetes from the color test done as part of this work reacted in the same way as the non-synaesthetes from Ward’s work. Teal and Yellow were rarely selected for the lower tones in the octave, while maroon and blue were rarely selected for the higher tones (Figure 3). This data leads to the idea that as pitch increases, colors matched to the tone will become brighter, and as pitch decreases, colors matched will become darker.

4. Design Process

The concept for a system of illuminating the music of a stringed instrument directly in the body of the instrument began to solidify during the research process. The research process included: evaluating data taken from the color surveys, investigating the phenomenon of synaesthesia, the fundamentals of musical theory, ambient peripheral displays, and the work of the pioneers of visual music.

It was decided that a violin would be a good choice as the instrument for the music illumination system because of its nature as a solo instrument, typically playing single notes. Whether by itself on the stage, or backed by a full orchestra during a concerto, the violin stands out. Also, it is possible to play any pitch between 196 Hz to ≈ 2100 Hz because the violin, along with other members of the string family, is a non-fretted instrument. The violin is also the smallest member of the string family, making production a manageable task.

The shape of a traditional violin has been well developed for acoustic and visual appeal, but an addition of an electronic pickup allowed exploration of a new form. A strong emphasis was placed upon a contoured form, especially where physical interaction occurs with the instrument. The rounded shape of a traditional violin creates an acute pressure point on the neck of musicians, felt mostly upon readjustment of the instrument. The form (Figure 4) was developed to minimize this pressure point with a contoured surface matching the curve of the human neck.

Figure 4. First Sketches of the Developed Form and Scroll, Plus a Full-size Sketch Model

Once the two-dimensional form had progressed, full-size three-dimensional foam models were constructed to test for comfort, proportions and scale. These rough models led to small tweaks to further refine the shape.

Once the foam models were created, the neck and scroll were given greater attention. More sketches were done to develop a scroll that would match the aesthetic of the body. The final scroll design retained the traditional stop used by violinists as a reference point, but removed excess weight from the end of the instrument.

As the scroll was being developed, careful consideration was given as to the best way to display the music illumination system. The lights needed to glow rather than to shine, and it was also extremely important that the lights not be so bright that they would be a distraction to the violinist. Another factor that was considered important was creating an instrument that would have a sophisticated aesthetic. The best way to incorporate all of these needs into the instrument was to have a wooden top and bottom, but to create the sides out of frosted acrylic. The frosted acrylic would serve as a light diffuser, which would give the instrument its glow, and since the light would be predominately exiting the side of the instrument, the violinist would not be distracted. The wood top and bottom would invoke a strong link to a traditional violin, but the shape and acrylic sides would separate it as a distinct instrument (Figure 5).

Figure 5. Early Sketch of the Side Profile of the Instrument

5. Design Results

The Vivisi electric violin (Figure 6), as the new instrument became known, is a musical instrument that produces a visualization of sound within the body of the instrument. Based on research of the medical phenomenon of synaesthesia, the Vivisi electric violin is able to display the rhythm, pattern and volume of music, to be an integrated visualization of sound for the hearing impaired.

The form of the violin is directly related to the comfort of the musician. A strong emphasis was placed upon a contoured form, especially where physical interaction occurs with the instrument.

The asymmetric form of the instrument gives visual interest, but also creates a longer viewing surface for the color to be displayed upon, as well as adding extra space for the on-board electronics. The “f” holes are reminiscent of the traditional “f” holes of a classical violin, but with a simpler, yet stronger visual dynamic.

Figure 6. The Vivisi Electric Violin

6. Reactions and Feedback

Once the prototype was developed to a certain degree of sophistication, it was possible to demonstrate the music illumination system to individuals with little or no hearing capabilities. Informal interviews were held with four deaf and hard of hearing adults, along with four hearing adults who frequently interact with deaf and hard of hearing individuals in non-hearing environments.

As was expected, there were some difficulties that arose when trying to explain the music illumination system to the hearing impaired interviewees. Part of the problem was centered on language. Understandably, musical terms were not particularly familiar to the deaf and hard of hearing individuals that were interviewed. Possibly because of the disconnect between the hearing and non-hearing worlds, or possibly because of the language barrier, the most profoundly deaf people that were interviewed saw the idea as interesting, but suggested that people with more hearing abilities would enjoy it more. This proved to be true. Individuals that were not profoundly deaf were more likely to understand the concept quicker and indicate a greater amount of interest in seeing a concert featuring the music illumination system.

The compelling aspect of the music illumination system is not that it creates a visual environment that is like music. It is that it creates a visual environment that is musical in nature. Music, by definition, is auditory. But at the core of music, fundamental concepts can be found such as rhythm, patterns and dynamic changes, which are not exclusively auditory concepts.

Conversations with the hearing adults that interact with deaf and hard of hearing individuals on a daily basis generated a high level of interest in the music illumination system. From these interviews a greater insight into the hearing impaired community was gained. Some of the hearing interviewees suggested that the music illumination system could be taught to children to convey mathematical concepts found in musical patterns and rhythms. Others suggested that the system might also be applicable to individuals with visual impairments, since perception of light would still be possible. In general, the idea of the music illumination system was embraced as benefiting the hearing and hearing impaired by enhancing the auditory experience by adding visual cues.

7. Future Implications

By creating an instrument that made performing both auditory and visual music possible, audiences with hearing capabilities and audiences with little to no hearing capabilities can be engaged simultaneously.

Also, the technology used to make the Vivisi a compelling performance instrument can be changed slightly to create a helpful training tool for students of the violin. There is vast potential for a tuning illumination system, designed to provide positive peripheral feedback to students when they are playing in tune. The tuning illumination system could be removed from the body of the violin and encased in an attachment to a music stand. This product could have great appeal to students of any stringed instrument, but also other instruments where a physical action is required to tune a note, such as a trombone. The attachment would be an ambient display of real-time tuning, possibly glowing on the left side for flat notes, on the right side for sharp notes, and glower brighter in the center for notes that are played in tune. The peripheral nature of the device would allow students to focus on sheet music, while giving gentle hints to help tune a current note. A tuning illumination system attached to a stand could be sold for a much lower price than an instrument, making the technology available to a wider audience.