Solution Overview: This demo creates an AI generated audio-reactive animation during a live musical performance. Each performance is different, starting with a single source image and building out into a ~1 minute mirroring animation over the course of the musical piece.
A demo DJ has a control panel that monitors the process and can actively influence the AI generation, change prompts, introduce noise or other effects into the img2img pipeline, control the audio-reactivity and live-cross with Synesthesia music visualization software.

Situation: With emerging development and importance of generative AI in 2023, showing impressive "AI on Intel" became a key message at events. We had an already established practice of featuring a musician at events and accompanying their performance with off-the-shelf audio visualizer software.
‍
Task: The demo team received a direct challenge / request from the Intel Events team for AWS re:Invent to see if we could use generative AI to drive a captivating live visual experience that accompanies the musician.

Process: Intel product teams had a clear "go to market" strategy for AI acceleration and messaging. I aligned closely with these teams to insure the final result would be a winning Intel AI performance demo. After clarifying the demo problem and goals, I started brainstorming and iterating on prototypes, starting with NVIDIA GPUs to establish a quick functional AI prototype, then migrating to Intel accelerated builds to run the AI on the A770 GPU. I worked closely with Intel hardware acceleration engineers to achieve the best available acceleration.
‍
Technical Problem Solving: To create a smooth animation from discrete AI-generated images, I decided to use an img2img pipeline, starting with a source image that would slowly morph into something else over several small iterations. The software would apply small transformations to the previous frame and feed the image back into the AI generation.

Even with GPU acceleration, the image generation was slower than real-time. To make the demo run and grow smoothly, frames were cached in TouchDesigner and played through like a deck of cards at ~ 25 fps.

The playback speed and a particle simulation pulsed with audio hits identified from sound signal processing. Because we couldn't rehearse with the performer in advance, I used audio recordings.  

Throughout the process, I had regular check-ins with event stakeholders, and built prototypes in our demo studio for executive and event team review.

The animation output was live-crossed with output from Synesthesia audio-visualizer software, allowing the DJ to focus more on abstract visuals at the beginning of the performance (when the AI animation was still relatively short) and also as a fallback if there was a problem with the AI generation.

Content Selection: For each event, Intel event PMs along with Intel brand teams had aligned on event visual language and theme. The theme guided decisions on the source images and AI prompts for each show.

Challenges:
It was critical for the "AI on Intel" performance story that this demo run live. Pre-processed AI was not a viable option. Planning several backup systems and fallback options for the live performance was an important part of the overall design.

StableDiffusion's native resolution is 512x512px, too low for the large screens we wanted to use. I used a combination of up-scaling, and mirroring the images. This created a kaleidoscopic visual effect at the edges.

To make the whole experience audio-reactive, I used both the live crossfade with Synesthesia, and a particle simulation reacting to beats. Audio processing was handled by Synesthesia and sent to TouchDesigner via OSC.

Results / Impact: This demo was a visual highlight at AWS Re:INVENT 2023, and based on it's success at drawing crowds and starting conversations, it took central importance at Intel’s CES 2024 demo room. Versions of this demo went on to be showcased at a POTUS host event, Intel’s booth at Computex and others. Below is a screenshot from Event Marketer.