I reported in November that Microsoft was about to start a bigger effort to bring AI features to Xbox, and today, the company is unveiling what it’s calling a breakthrough in AI for gaming. Microsoft’s new Muse AI model could help Xbox developers create parts of games in the future, and the company says it’s even exploring the potential of using it to preserve classic games and optimize them for modern hardware.
Microsoft Research has created Muse, a first-of-its-kind generative AI model that can generate a game environment based on visuals or players’ controller actions. It understands a 3D game world and game physics and can react to how players interact with a game.
“This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators,” explains Fatima Kardar, corporate vice president of gaming AI at Microsoft.
The Muse model was trained on a large amount of human gameplay data from the Xbox game Bleeding Edge thanks to a collaboration between Microsoft Research and Xbox studio Ninja Theory — both of which are based in Cambridge in the UK. “This partnership allowed us to closely collaborate with the game studio to understand what needs to be in place and how we can responsibly unlock access to a large amount of gameplay data,” says Katja Hofmann, head of Microsoft Research’s game intelligence team.
This allowed Muse to access the equivalent of seven years of human gameplay, resulting in the model being trained on a billion image action pairs in total. It’s a milestone research project that’s being published in Nature today, but as it’s early work, the model is limited to generating gameplay visuals at a resolution of just 300×180 pixels. That’s up from the 128 x 128 resolution of Microsoft’s earlier work on generative AI gaming models, but it’s still far behind the 1080p (1920 x 1080) resolution that is common among PC gamers.
Microsoft has released a number of examples of the Muse model in action, generating gameplay and even allowing players to load visual elements into the game to prompt the model. This could be used for the early iteration stages of a game, but Microsoft is stressing this isn’t designed to generate an entire game and replace creators.
“For Xbox, the principles that drive our approach to AI are to create more value for players and game creators, bring more games to more people around the world, and recognize that the development of a great game will always be grounded in the creator’s vision and artistry,” says Kardar. “We believe generative AI has the potential to enhance this creativity and unlock new possibilities.”
During a press briefing, Hofmann also demonstrated a real-time version of Muse that Microsoft is currently experimenting with to enable interactive AI-powered games. The demo generated game visuals on the fly in real time and even reacted to objects being dropped into the game to change the environment. While the real-time gameplay only ran at 10fps and a 300×180 resolution, it was an early demonstration of what might be possible in the future.
Microsoft is now exploring how Muse could help improve classic games and bring them to modern hardware. “You could imagine a world where from gameplay data and video that a model could learn old games and really make them portable to any platform where these models could run,” says Microsoft Gaming CEO Phil Spencer. “We’ve talked about game preservation as an activity for us, and these models and their ability to learn completely how a game plays without the necessity of the original engine running on the original hardware opens up a ton of opportunity.”
Microsoft is also exploring how Muse can help game developers prototype games or how teams could add new AI-powered experiences to existing titles. Some short interactive AI game experiences will be available on Copilot Labs soon for people to try out. Microsoft is also planning to share AI tools and experiments with Xbox players and creators “earlier on” so that it can “make sure that our AI innovations address real problems and add new value to creating or playing with Xbox.”
While Microsoft is still in the exploration phase of AI in gaming, it’s also making it clear it’s up to individual studios at Xbox how they want to use AI in games. “As part of this, we have empowered creative leaders here at Xbox to decide on the use of generative AI. There isn’t going to be a single solution for every game or project, and the approach will be based on the creative vision and goals of each team,” says Kardar.
There is already a lot of fear from game developers and studios about how AI could affect the creative process of making games, particularly at a time when barely a week goes by without layoffs in the gaming industry. A recent report revealed that 1 in 10 game developers lost their jobs in 2024 alone.
“We don’t intend to use this technology for the creation of content,” says Ninja Theory studio head Dom Matthews. “I think the interesting aspect for us that’s exciting, is how can we use technology like this to make the process of making games quicker and easier for our talented team, so that they can really focus on the thing that’s really special about games: the human creativity.”
