Goodbye is a live vocal performance of an originally written song featuring real-time augmented reality (AR) and handtracking. The handtracked real-time augmented reality (AR) effects illuminate the act of waving goodbye. The handtracked AR manipulates the live music while the live music also influences the AR visuals in real-time. This back and forth dance creates a unique performance and combination of instruments.
As a musician dabbling in AR/VR, I wanted to combine these creative mediums together and explore the possibilities of AR in live music performances, and potentially influencing my artistic practice.
This live performance was a 2-day performance with an audience of about 100 people each day. The performance was created with the venue and audience in mind.
DECIDING THE INPUT
Hand tracking: I used Meta Spark to create an AR instrument that uses hand tracking to control the sound while the sound controls the scale and rotation of the “performing kirby” model.
Try out the AR instrument:
Physical controller: Potentially building a controller to control music in Ableton. I tested this interaction using an Arduino’s accelerometer and mapped it to different Ableton controls.
I received feedback that using handtracking for a performance would feel very magical and that the seamlessness of using hands to control sound incorporated with AR would be extremely innovative. Hence, I decided on handtracking as my performance input.
I used Meta Spark to create a first prototype. I wanted to explore using AR to affect the music more but was unsure how to prevent the song from being too distorted and still have some form of control over the music. Hence, for the first prototype, I focused on having the music affect the AR.
I consulted some experts in music technology and they suggested I use AR to vary the delay, echo, reverb, pitch, tembre, volume to keep the song structure while allowing AR to affect the music more. I performed this initial prototype in front of an audience and I asked them on their thoughts if they would prefer seeing how the technology works (i.e. what is affecting what) or leaving it up to interpretation. The majority felt like it was more magical not knowing what is affecting what but still being able to hear or see the impact of it.