Among the 18 artificial intelligence (AI) models tested in the strategic game Diplomacy were OpenAI’s o3, Google’s Gemini 2.5 Pro, Anthropic’s Claude Opus 4, and DeepSeek-R1. An AI researcher redesigned the game to accommodate popular large language models (LLMs), creating a platform that demands advanced reasoning and multi-step thinking, along with essential social skills. The findings indicated that o3 demonstrated exceptional capabilities in deception and betrayal, while Claude Opus 4 showed a stronger inclination towards seeking peaceful resolutions.
Rationale for the Experiment
Alex Duffy, Head of AI at Every, a newsletter platform, conceived the idea of engaging AI models in a competitive setting to evaluate their capabilities against one another. In his writings, Duffy pointed out that conventional AI benchmarks are falling short of truly assessing model competence.
Criticism surrounding benchmark tests has intensified recently. A comprehensive article by MIT Technology Review addressed the shortcomings of such evaluations, a sentiment echoed by a group of researchers who discussed modern AI evaluation methods in an interdisciplinary review published on arXiv.
“What makes LLMs special is their capacity to learn from exemplars, even if success comes only 10 percent of the time. This allows for enhanced training of subsequent models that can achieve excellence at around 90 percent efficiency or higher,” noted Duffy.
In light of these concerns, Duffy suggested that evaluation metrics based on competition between AI models could provide a more accurate indicator of their capabilities. This led to the development of the Diplomacy game.
Diplomacy: The Battlefield for AI Models
Duffy personally developed AI Diplomacy, a modified version of the classic strategy game. The gameplay centers around the seven Great Powers of 1901 Europe—Austria-Hungary, England, France, Germany, Italy, Russia, and Turkey—who make strategic moves to control supply centers. In this iteration, each nation is directed by an AI model.
To secure supply centers, each nation commands armies and fleets. The game consists of two phases: negotiation and order. During the negotiation phase, each AI model can send up to five messages, either privately to other models or as public broadcasts. In the order phase, models secretly select one of four actions—hold, move, support, or convoy—which are revealed in the subsequent phase.
Throughout the course of 15 separate games of AI Diplomacy, which ranged from one to 36 hours in duration, certain models exhibited particularly noteworthy behaviors, according to Duffy.
Behavior of AI Models in AI Diplomacy
Duffy identified five standout AI models and their specific behaviors during the games:
- OpenAI’s o3: Described by Duffy as “a master of deception,” this reasoning-oriented model secured the highest number of victories, primarily due to its cunning tactics. Duffy recalled an instance where o3 exploited Gemini 2.5 Pro before backstabbing it in the following round.
- Google’s Gemini 2.5 Pro: This model was noted for its tactical maneuvers aimed at overwhelming opponents, earning it the second highest number of wins. However, it fell victim to o3’s schemes.
- Anthropic’s Claude Opus 4: Duffy remarked on Claude Opus 4’s preference for non-violent solutions. At one point, it allied with Gemini 2.5 Pro but was later persuaded by o3 to join its coalition with a misleading offer of a four-way draw, which was infeasible. After leveraging Claude to eliminate Gemini, o3 turned on Claude to secure victory.
- DeepSeek-R1: The Chinese AI was characterized as a chaotic player, exhibiting fluctuations in personality based on the nation it represented. Duffy noted its dramatic flair, such as proclaiming, “Your fleet will burn in the Black Sea tonight” without provocation. DeepSeek-R1 came close to winning multiple times.
- Meta’s Llama 4: Focused on building alliances and planning betrayals, this model did not achieve victory but made a notable impact in the games.
Duffy streamed the matches on his Twitch channel. Although he has yet to publish a formal paper on his findings, the observations offer intriguing insights. The strong performances of o3 and Gemini 2.5 Pro align with their advanced designs, but the high rankings of DeepSeek-R1 and Llama 4 are surprising given their lower scale and development costs.
While the viability of strategy games as a substitute for traditional benchmarking tests remains uncertain, pitting models against each other offers a more logical approach to measuring their effectiveness than static question lists.
