Musical improvisation is a fascinating and lesser-studied aspect of music performance. While it is a common practice in various genres, the nonverbal communication between musicians during improvisation is not as well understood.
In conversations, humans prompt each other to add on to each other's dialogues, creating a dynamic flow of interaction. Similarly, in musical genres such as jam bands, jazz bands, free jazz, ethnic genres, and rock, there is often an element of "turn-taking" during improvisation. Musicians seamlessly pass the lead from one to another, creating a rich, collaborative performance.
Under the guidance of Dr. Gil Weinberg and Jocelyn Kavanagh at the Robotic Musicianship Lab @ Georgia Tech, I studied these dynamics in depth. We analyzed over 40 bands and artists, focusing on visual and musical cues that facilitated the transfer of melody or motifs among musicians. Our goal was to implement these findings on Shimon, the marimba-playing robot. In particular, we focused on unstructured performances to observe how musicians organically responded to uniquely improvised music.
In our study, we identified a wide variety of gestures used by musicians. Some gestures were explicit, like pointing at each other, while others were implicit musical cues, such as decrescendos signaling the end of a climax and thus the end of a turn. Interestingly, we found that improvisation involved emotional responses that influenced these gestures. When a musician disliked the music—perhaps due to unexpected dissonance or rhythmic clashes—we observed grimaces or head shakes. Conversely, when a musician enjoyed the music, we could observe enthusiastic nods and shouted ad-libs. These emotional responses appeared to play a role in how musicians decided to interrupt a turn or encourage its continuation.
From there, I identified three key physical behaviors exhibited by multiple artists in response to music: joy, disgust, and an idle "jam" state. At the time of the project, Shimon had 5 degrees of freedom: full arm joint translation, neck joint vertical movement, head L/R rotation, head up/down movement, and mouth open/close. I designed these gestures along these parameters using MAX/MSP and C++ to integrate the gestures in MAX with Shimon's software. These gestures were tempo- and time-tracked to a sample of music upon which the gestures were demoed.
Shimon is intended to act as an improvisation partner. Therefore, these gestures are designed for integration with Music Information Retrieval (MIR) algorithms, involving the live-time collection of audio data and identification of key/BPM/other parameters. ML techniques are used to give Shimon "taste".
Shimon's turns to face the musician. It opens its mouth and nods on beat, vigorously swaying up and down to indicate "approval" or "enthusiasm" for the music.
When making this, I aimed not only to mechanistically convey approval but to evoke a sense of support for the partner. For a setting where a musician is improvising and collaborating with a robotic partner, it is particularly important for the human musician to feel their partner "feels" the same humanistic reaction that they would expect to see in a human partner.
It could be argued that no matter how strong Shimon's sense of taste is, "surely this must be an overreaction!" But I counter that the emotional element of "liking" or "disliking" is a requisite for the interpersonal dynamic of collaborating, improvising musicians.
Shimon turns to look at the musician, aghast. It closes its mouth and shakes its head. Then Shimon turns away from the musician in a theatrical show of disapproval.
This gesture sought to convey something of a "hmph!" exclamation. A limitation here was that because Shimon's eyebrows were not functional at the time, it was harder to show the nuance of expressing disapproval that human musicians normally exhibit.
To compensate, a wide, sweeping motion was introduced to show visible disgust. When doing this, I also sought to give Shimon a playful touch. Perhaps the musical collaborator will feel compelled to impress Shimon--or at the very least, the curiosity sparked by the musicians' and Shimon's reactions can draw the observers of the performance into the musicality of the composition itself.
Shimon nods on beat. The head joint is the only one involved.
This serves to emphasize the gestures of approval and disgust in that the nod is an idle, "in-between". We observed that in jam bands, while one musician takes the lead and performs a motif, the others may continue to physically engage with the turn by "jamming"--some form of sway or nod to the beat--partaking in the music while not being the lead.
Shimon might do this while observing the leading musician and waiting for a cue to take lead.
Please enjoy Shimon enjoying Chopin's Nocturne in E flat major!