Over the weekend, Meta announced the release of two new Llama 4 models: a compact version known as Scout, and a mid-size model named Maverick. The company asserts that Maverick surpasses competitors GPT-4o and Gemini 2.0 Flash on a variety of key benchmarks.
Maverick quickly claimed the second position on LMArena, an AI benchmarking platform where users assess and vote on the performance of various models. In a press release, Meta showcased Maverick’s ELO score of 1417, placing it just above OpenAI’s 4o and below Gemini 2.5 Pro, indicating that Maverick triumphed more often in head-to-head matchups with other systems.
This development positioned Meta’s Llama 4 as a serious competitor against the leading closed models from OpenAI, Anthropic, and Google. However, a closer examination of Meta’s documentation revealed an unexpected detail.
Meta admitted in fine print that the version of Maverick featured on LMArena differs from the publicly available model. The company clarified that the LMArena version was an “experimental chat model” specifically optimized for conversational performance, a detail previously reported by TechCrunch.
LMArena expressed disappointment regarding Meta’s interpretation of its benchmarking policies, stating, “Meta should have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a customized model to optimize for human preference. We are updating our leaderboard policies to ensure fair and reproducible evaluations to avoid such confusion in the future.”
As of the publication of this article, a representative from Meta had not provided a response to LMArena’s concerns.
While utilizing an optimized version of Maverick for LMArena testing does not explicitly violate the site’s rules, LMArena has previously raised alarms about the risk of “gaming the system,” taking measures to prevent overfitting and benchmark leakage. The practice of submitting specially tuned models while releasing different versions to the public undermines the validity of benchmark rankings.
Independent AI researcher Simon Willison remarked, “It’s the most regarded general benchmark because all of the other ones suck. When Llama 4 debuted, I was genuinely impressed that it ranked second in the arena, just behind Gemini 2.5 Pro. I regret not reading the fine print closely.”
Following the release of Maverick and Scout, conversations emerged in the AI community about potential speculation that Meta had trained its Llama 4 models to excel on benchmarks while concealing their actual limitations. Addressing these claims, Ahmad Al-Dahle, Meta’s VP of generative AI, stated, “We’ve also heard claims that we trained on test sets – that’s simply not true. We would never undertake such practices. The variability in performance is likely due to the need to stabilize implementations.”
“It’s a very confusing release generally.”
Notably, observers pointed out that Llama 4 was unveiled during an unusual weekend timeframe, which is atypical for major AI announcements. When prompted on Threads about the timing, Meta CEO Mark Zuckerberg responded, “That’s when it was ready.”
Reflecting on the situation, Willison concluded, “This entire release is very confusing. The model’s score is essentially worthless to me; I can’t even access the version that achieved the high ranking.”
Meta’s journey in bringing Llama 4 to the public was not without challenges. A recent report from The Information indicated that the launch faced numerous delays due to the model’s failure to meet the company’s stringent internal expectations, particularly following the buzz surrounding DeepSeek, an open-source AI startup from China.
Ultimately, the use of an optimized model like Maverick in LMArena presents a complicated scenario for developers. When they evaluate models like Llama 4 for their applications, they tend to rely on benchmarks for insights. However, the benchmarks may highlight capabilities that are not available in the versions meant for public use.
This incident underscores the growing importance of benchmarks in the rapidly evolving field of AI, as well as Meta’s drive to be recognized as a leader in this competitive landscape, even if it raises questions about the integrity of benchmarking processes.
