On Wednesday, Microsoft researchers unveiled a groundbreaking foundation model known as Magma, capable of executing agentic functions. This advanced artificial intelligence (AI) model has been pre-trained on a vast array of datasets that encompass text, images, videos, and spatial formats. The company from Redmond emphasized that Magma builds on existing vision-language (VL) models, enabling it to process multimodal information and perform planning and actions based on that information. Its versatile capabilities make it suitable for various applications, including computer vision, user interface (UI) navigation, and robotic manipulation.
Microsoft Introduces Magma Foundation Model
In a detailed post on GitHub, Microsoft researchers elaborated on the features and functions of the new Magma foundation model. Unlike conventional models that are often distilled from existing frameworks, foundation models like Magma are developed entirely from the ground up and can serve as the foundational layer for subsequent models. Its extensive pre-training on diverse datasets differentiates Magma from many of its predecessors.
The underlying architecture of Magma is based on the Llama 3 AI model. However, Magma enhances this framework with capabilities to plan and act within visual-spatial environments. This advancement not only allows the model to generate responses similar to a chatbot but also enables it to carry out actions.
When paired with camera sensors, Magma has the potential to function as a computer vision chatbot, providing insights about its surroundings. Additionally, it can manipulate the user interface of devices and, more intriguingly, control robots to execute complex tasks through its agentic capabilities.
Researchers at Microsoft attributed the model’s advanced functionalities to its comprehensive dataset and two key technical elements—Set-of-Mark and Trace-of-Mark. The Set-of-Mark component aids in action grounding within images, videos, and spatial data by allowing the model to predict numerical markers for buttons or robotic arms in visual contexts. Meanwhile, the Trace-of-Mark component incorporates temporal video dynamics, enabling the model to forecast subsequent frames prior to taking action, which enhances its spatial comprehension.
In internal assessments, Microsoft researchers reported that Magma attained competitive scores in various agentic evaluation tests, surpassing models from notable companies such as OpenAI, Alibaba, and Google. Currently, the company has not made Magma available to the general public.
