Live AR interaction for virtual presentation
Situation: During CEO Pat Geslinger's virtual keynote presentation at Innovation 2021, he planned to discuss changes in upcoming transistor designs. His keynote team asked the demo team to prototype virtual whiteboarding or an interactive overlay that could be used in the live feed, so that Pat could discuss the changes and benefits of the new transistor design while speaking directly to the audience.
Task: I had already been prototyping several interactive overlays for video calls, so I had a prototype to share during an early brainstorming session. Based on this, I was tasked to build the demo while aligning closely with the event producer, a 3D artist, an external agency working on keynote graphics, and content experts from the Technology Development business group.
Process: The AR overlay effect was accomplished with body tracking software and a second screen just off camera, allowing the presenter to see the AR overlay and guides, and placed so it looked like the presenter was looking at the 3D object.
Based on the keynote script, we knew what 3D models would be needed, and key transitions / animations that we'd have to use, as well as opportunities for AI interaction. The AR overlay let the presenter use their hand to rotate the a 3D model for the audience, and hidden targets would trigger an advance in the animation.
We had a video animation as a fallback in the live AR was a no-go for whatever reason, and received the final buyoff from the Keynote team and Pat during his pre-event executive rehearsal, where I walked him through the AR overlay and how to control it.
We only got two rehearsals with Pat, so user information design and redundant backups were a huge part of the overall design. In addition to a compete backup on a KVM switch backstage, we also had the ability to remotely trigger animations, and a tablet interface ready so that Pat himself could manually jump to a certain part of the animation if needed.
Here's a shot of the setup. The software, built in TouchDesigner, ingests the house camera feed (camera in upper left,) and creates two output displays. The first is a clean feed of the source video with the AR overlay. The second display (right) shows the presenter the AR overlay, along with visual guides showing where to interact with the part and how to advance the animation.
Impact: The demo was a successfully run during the virtual keynote of Innovation 2021, the event producers were thrilled, and it contributed to the overall success of the event.
