On Wednesday, Microsoft unveiled an innovative artificial intelligence (AI) model designed to create 3D gaming environments. Known as the World and Human Action Model (WHAM) or Muse, this new AI initiative is a product of collaboration among Microsoft’s Research Game Intelligence, Teachable AI Experiences (Tai X), and Xbox Games Studios’ Ninja Theory. The company indicated that this large language model (LLM) aims to assist game designers in the creative process and streamline the generation of game visuals and controller actions for developers.
Microsoft Unveils Muse AI Model
In a blog post, Microsoft provided insights into the capabilities of Muse. Although still a research prototype, the company announced plans to open-source the model’s weights and sample data for the WHAM Demonstrator, which serves as a conceptual interface to interact with the AI. Developers can access the model via Azure AI Foundry, and a research paper outlining the technical details has been published in the journal Nature.
Training a model to navigate such a complex domain presented significant challenges. Microsoft researchers gathered extensive gameplay data from the 2020 title Bleeding Edge, published by Ninja Theory. The LLM was trained using a billion image-action pairs, roughly equivalent to seven years of gameplay experience. The data was sourced ethically and is intended solely for research applications.
The process of scaling the model for training proved to be a major hurdle. Initially, Muse was trained on a cluster of Nvidia V100 GPUs before transitioning to a more robust setup using multiple Nvidia H100 GPUs.
In terms of functionality, the Muse AI model can process both text prompts and visual inputs. Once a gaming environment is created, users can enhance it further through controller actions. The AI adapts to user movements to generate new environments that align with the initial prompt and maintain consistency within the gameplay experience.
Given its unique nature, the model poses challenges in standard benchmark testing. Researchers noted that they have conducted internal assessments based on metrics such as consistency, diversity, and persistence. While still in the research phase, the outputs of the model are currently limited to a resolution of 300x180p.
