At the 2024 MTV Video Music Awards, Eminem performed alongside a digital version of his 1999 alter ego, Slim Shady. The live event at UBC Arena in New York was powered by Metaphysic’s AI technology, bringing a past version of the artist to the stage in real time. Metaphysic’s work wasn’t limited to the live performance—they also won the Best Visual Effects award for their role in creating Eminem’s Houdini music video.
The Workflow Behind a Live AI Event
The process of bringing Slim Shady to life was complex and required careful planning, according to Jo Plaete, VFX Supervisor at Metaphysic. “The basic workflow involves several key stages, each tailored to address the unique challenges of the specific show,” Plaete explains. “Initially, we start with data collection and preparation, gathering high-quality reference images and videos of the person who will be represented as the AI avatar. These references include various facial expressions, angles, and lighting conditions to train our AI models effectively.”
Once this step is complete, the team moves on to training the AI models using deep learning. “We process and annotate the collected media, then use these algorithms to train the AI model to replicate the person’s facial features and expressions,” Plaete notes. This part of the workflow guarantees the avatar’s realism during the live event.
Preparing for the Performance
Plaete details how the Metaphysic team adapted their technology for the specific challenges of the VMA show. “We faced the challenge of multiple hybrid Eminem performers on stage, with potentially 30 to 50 people in the frame.” To address this, the teams prepared their system to manage the complexity of live installations, ensuring that everything would run smoothly when the cameras rolled.
Once at the venue, the team conducted a load-in and hardware setup, testing the signal feeds to ensure everything functions as expected. “Rehearsals are a critical part of our workflow,” Plaete notes. “They allow us to tune the technology, capture rehearsal footage, and feed that data back into the training of our models.”
Real-Time AI in a Multicamera Broadcast
One of the critical aspects of Metaphysic’s technology is its ability to generate the AI avatar on the fly and integrate it into a multicamera broadcast without introducing latency. According to Plaete, “The avatar is generated via an optimized processing pipeline that leverages the fastest available GPUs and specialized hardware.” This ensures that the AI rendering remains fast enough to meet the demands of a live event.
Collaboration with the venue and broadcast team proves essential. “We work closely with the broadcast team to ensure that the signal pipeline—from camera to our processing and back to the broadcast truck—does not introduce any unnecessary latency.”
Markerless Facial Tracking
A unique aspect of Metaphysic’s technology is that it doesn’t require special markers or tracking aids for the stand-in performer. “Our system uses advanced computer vision algorithms to automatically detect faces in the video feed,” Plaete says. “The talent can focus entirely on their performance, and the technology handles real-time tracking.”
For the VMA, a stand-in takes on the physical movements of Slim Shady, while the AI model applies the face in real time for both the broadcast audience and the live crowd. This creates a believable recreation of Slim Shady without distracting the performer or the audience.
What’s Next for AI in Live Events?
Metaphysic is pushing the boundaries of what’s possible in live AI-driven performances. “We are constantly looking to enhance the realism and quality of our work,” says Plaete. “With our expert eye, we see opportunities to improve our models, refine our integration processes, and potentially run the system even faster.”
“We see the possibility of extending this technology to swap the entire body or create interactive experiences incorporating AI-generated backgrounds and lighting effects,” Plaete adds. “The possibilities are vast, and we believe this technology can evolve in ways that truly change how live experiences are created.”
A New Era of Visual Effects
Metaphysic’s collaboration with production company Synapse and director Rich Lee on the Houdini music video demonstrates the potential of AI in visual effects. In the video, Eminem’s alter ego, Slim Shady, returns to step out of a portal from the past, with Metaphysic’s AI recreating his youthful look from the late 90s. The result is a visual effect that feels natural and authentic, allowing both Slim Shady and Marshall Mathers to be believable on-screen.
The success of the Houdini project, completed in just under three weeks, isn’t only due to the technology but also the collaboration between the AI-generated visuals and human performances. The video went viral, garnering millions of views and earning Metaphysic widespread acclaim for their technical achievement.
For more details on Metaphysic’s work and their AI-driven tools, please visit www.metaphysic.ai.
