On Monday, Microsoft researchers introduced a groundbreaking artificial intelligence (AI) system that outperforms human doctors in diagnosing patients. Named the Microsoft AI Diagnostic Orchestrator (MAI-DxO), this system incorporates multiple AI models within a framework designed to analyze patient symptoms and medical history, leading to pertinent test recommendations. Following the tests, it suggests potential diagnoses while emphasizing cost-effectiveness in the testing process.
Microsoft Develops Benchmark to Test MAI-DxO’s Performance
Mustafa Suleyman, CEO of Microsoft AI, shared an update about MAI-DxO in a post on X (formerly Twitter). He characterized the system as a significant leap toward medical superintelligence, capable of addressing some of the most challenging medical cases with improved accuracy and reduced costs in comparison to conventional diagnostic techniques.
The MAI-DxO operates as a virtual panel comprising physicians with varying diagnostic methods, collaborating to address medical challenges. Microsoft described the system in a blog post as utilizing a multi-agent framework, where one agent proposes a hypothesis, another selects the tests, two provide checklists and oversight, and the last one challenges the proposed hypothesis.
MAI-DxO workflow
Photo Credit: Microsoft
Once a hypothesis is validated by the panel, the AI can either pose a question, request further tests, or deliver a diagnosis if it deems the data sufficient. When recommending tests, the system conducts a cost analysis to maintain reasonable overall expenses. Notably, the system is model agnostic, meaning it can integrate with various third-party AI models.
Microsoft asserts that MAI-DxO enhances the diagnostic capabilities of every AI model assessed, with OpenAI’s o3 achieving the highest accuracy by correctly diagnosing 85.5 percent of cases in the New England Journal of Medicine (NEJM) benchmark study. In comparison, 21 practicing physicians from the US and UK, each with five to 20 years of clinical experience, managed only a 20 percent accuracy rate in the same cases.
The company explained that MAI-DxO can operate under predefined cost limitations. By inputting a budget, the system evaluates cost versus value in its diagnostic decisions, ensuring that only necessary tests are ordered, rather than conducting exhaustive testing for every potential cause of the symptoms.
To evaluate the performance of the AI system, Microsoft also created a new metric called the Sequential Diagnosis Benchmark (SD Bench). Unlike traditional medical benchmark assessments that rely on multiple-choice questions, this benchmark examines the AI’s ability to ask pertinent questions iteratively and order appropriate tests. The findings are then compared against outcomes published in the NEJM.
It is important to note that the MAI-DxO is not yet approved for clinical application and is currently part of early research aimed at enhancing AI’s role in diagnostic processes. Microsoft emphasized that for clinical use approval, the AI system must undergo thorough safety testing, clinical validation, and regulatory scrutiny.
