As part of our ongoing exploration into the fusion of music and visual art, we recently collaborated with Kumanta, a band project from PXL Music, to create a compelling audiovisual show. The goal was to enhance their live performance with real-time, interactive visuals tailored to each song’s mood and narrative. Across five tracks, we employed different technologies—ranging from point clouds to motion capture avatars—to elevate the emotional impact and audience engagement.

1. Visuals that Move with the Music

For two songs in the set, we focused on capturing live motion. We positioned an Azure Kinect camera at the lead singer to generate real-time point cloud data, which was then processed in TouchDesigner. The resulting visuals weren’t static backgrounds—they moved and pulsed with her every motion, transforming her presence into swirling particle effects that brought a living, breathing energy to the stage. This interaction blurred the lines between the physical and the digital, turning the singer into both performer and visual source.

2. From Reality to Anime: Embodying the Singer as an Avatar

For two other songs, we transformed the lead singer into a stylized anime avatar. Dressed in a Rokoko motion capture suit, she performed live while her digital counterpart mimicked every movement on screen. The avatar itself was designed in VRoid Studio, customized to reflect her visual identity. We then imported the character into Unreal Engine 5, where a real-time render was streamed via NDI cameras into Resolume for final visual treatment. This hybrid setup allowed us to create a stylized, yet expressive digital version of the performer—perfectly synced with her movements in real-time.

3. The Final Touch: Live VJing

While the reactive and mocap visuals added narrative and technological depth to the show, the experience was tied together by a live VJ, who layered in additional visual effects during the performance. This final touch ensured that the visuals didn’t just follow the performers—they also responded to the atmosphere of the room, making each show a one-of-a-kind experience.