Mark Zuckerberg recently released a video of himself using a leg press machine at a gym on his personal Instagram. However, this wasn’t just any regular workout clip - the leg press machine and its surroundings transformed into several imaginative versions throughout the video, including a neon futuristic gym, an ancient Roman setting, and even a gold-flaming version.
The workout clip turned out to be more than just a display of fitness. Zuckerberg introduced Movie Gen - Meta’s new GenAI model family that enables users to create and edit video and audio from simple text prompts. Going beyond creating and editing text-to-video clips, Movie Gen allows users to edit existing clips, like adding an object into someone’s hand or changing the background.
Movie Gen builds on Meta’s previous GenAI models including the Make-A-Scene, which facilitates the creation of various types of media, and the Llama Image foundation models, which enhance video and auto quality.
The newest addition to Meta's GenAI suite, Movie Gen is the third wave, and it features four primary capabilities - video generation, personalized video generation, precise video editing, and audio generation. According to Meta, Movie Gen models are trained on a combination of publicly available and licensed datasets.
The Movie Gen model is optimized for both text-to-image and text-to-video and is based on a 30-billion parameter transformer capable of creating videos up to 16 seconds long at a rate of 16 frames per second. In contrast, the Audio Gen video-to-audio model is built on a 13-billion parameter transformer.
The Personalized Video model allows users to upload an image and combine it with a text prompt to create customized videos that feature the individual or object in various scenarios. Meta explained that the same foundational transformer model powers the Precise Video Editing model, which allows users to edit existing videos by simply uploading the video and providing a text input describing the desired changes.
Movie Gen is still a work in progress, and Meta has not shared an official release date. However, the model is expected to be released next year. So far, Movie Gen has only been tested by a few filmmakers to get feedback. The company generally opts to open-source its AI innovations, as it has done with its Llama models, so developers may soon get the opportunity to experiment with Movie Gen.
Meta also released a research paper that highlights major breakthroughs achieved with Movie Gen for text-to-video generation, video personalization, and audio generation. The paper shares technical details of the model architecture, inference optimizations, and evaluation protocols.
According to the findings presented in the research paper, Meta claims that Movie Gen outperforms its competitors in audience ratings across several attributes, including consistency and realism. The model was tested against rivals such as OpenAI's Sora, Kling 1.6, LumaLabs, and Runway Gen3.
The paper also underscores the need for further research before real-world deployment, ensuring the system is safe from potential biases, unintended associations, and synchronization issues in generated content.
"Today, we’re excited to premiere Meta Movie Gen, our breakthrough generative AI research for media, which includes modalities like image, video, and audio," Meta stated in a blog post. "Our latest research demonstrates how you can use simple text inputs to produce custom videos and sounds, edit existing videos, and transform your personal image into a unique video. Movie Gen outperforms similar models in the industry across these tasks when evaluated by humans."
Meta emphasizes that while Movie Gen offers a wide range of use cases, however, it is not a replacement for the work of artists and animators. Meta plans on making further optimizations to decrease inference time and improve the quality of the videos it generates. The company is also working closely with filmmakers and creators to integrate their feedback.
Movie Gen represents a transformative step forward in video creation and democratizing the filmmaking process. The tool can enable a broader range of creators, making video production faster and more affordable. With Movie Gen, the possibilities for personalized and innovative content are virtually limitless.
