Sony is working on a prototype AI-powered version of at least one its PlayStation game characters. An anonymous tipster has shared an internal video from Sony’s PlayStation group with The Verge that demonstrates an AI-powered version of Aloy from Horizon Forbidden West.
The video is narrated by Sharwin Raghoebardajal, a director of software engineering at Sony Interactive Entertainment who works on video game technology, AI, computer vision, and face technology for Sony’s PlayStation Studios Advanced Technology Group. Raghoebardajal demonstrates an AI-powered version of Aloy that can hold a conversation with a player through voice prompts during gameplay.
Aloy can be seen responding to queries with an AI-powered synthesized voice and facial movements, both in a demo setting and within the full Horizon Forbidden West game. Raghoebardajal makes it clear this is just a prototype that has been developed alongside Guerrilla Games to demonstrate the technology internally at Sony.
The technology demo uses OpenAI’s whisper for speech-to-text, and both GPT-4 and Llama 3 for conversations and decision making. Sony’s has its own internal Emotional Voice Synthesis (EVS) system that it uses for speech generation, according to Raghoebardajal, and audio to face animation is powered by Sony’s own Mockingbird technology.
While the demo is seen running on PC, Sony has also experimented with running parts of this technology directly on PS5 consoles with “little overhead,” according to Raghoebardajal. Sony first demonstrated this internally a year ago, before showing a more advanced version behind closed doors at its Sony Technology Exchange Fair (STEF) in Tokyo in November.
“This is just a glimpse of what is possible,” says Raghoebardajal. This tech demo doesn’t answer the more obvious questions over whether it even makes sense to talk to Aloy when you’re supposed to be playing as her in Horizon Forbidden West, or the impact such a technology could have on voice actors and game developers.
Nvidia has been working on its own similar technology for AI-powered NPCs in games, where you speak freely to video game characters. Ace, as Nvidia calls it, has been demonstrated several times throughout 2024, and the GPU maker has even worked with Inworld AI to create Covert Protocol, a playable tech demo of Nvidia’s AI game technology.
Microsoft has also been partnering closely with Inworld AI to eventually bring AI characters to Xbox, allowing game developers to use generative AI characters, storylines, and more. Microsoft has also created its own Muse AI model that generates gameplay, designed initially for game developers to create a game environment for prototyping and ideating.
There is already a lot of fear among game developers and studios about how AI could impact the creative process of making video games, particularly when so many layoffs are impacting the gaming industry. Almost half (49 percent) of 3,000 respondents to a Game Developers Conference (GDC) 2024 survey said that generative AI tools are currently being used in their workplace, with 31 percent saying they personally use them.
Sony’s experiments with AI-powered PlayStation characters are bound to generate more conversation around the role of AI in video game creation, just ahead of GDC next week in San Francisco. We reached out multiple times to Sony for a comment on this leaked internal video, but the company didn’t respond in time for publication.
